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Summary REL 2010-No. 086 

A systematic comparison of the 
American Diploma Project English 
language arts college readiness 
standards with those of the ACT, College 
Board, and Standards for Success 



This study of four national English lan- 
guage arts college readiness standards 
sets compares content alignment and 
level of alignment of the standards 
statements in three comparison sets to 
a benchmark set, the American Diploma 
Project (ADP), and analyzes the cognitive 
complexity of all four sets. Standards 
statements in the comparison sets align 
completely or partially to varying propor- 
tions of the ADP benchmark's 62 stan- 
dards statements — 77 percent for the 
College Board College Readiness Stan- 
dards, 68 percent for Standards for Suc- 
cess, and 34 percent for the ACT College 
Readiness Standards. But only 5 percent 
of the ADP statements completely align 
with content in all three comparison sets, 
a share that rises to 27 percent when 
partial alignment is also considered. A 
majority of statements in the four sets 
(53-68 percent) were rated level 3 on a 
four-level cognitive complexity scale. 

The country’s interest in college readiness has 
intensified in recent years. Four sets of English 
language arts college readiness standards — 
content statements specifying what students 



should know and be able to do to succeed 
in entry-level college courses — intended for 
national use have been developed in the past 
decade. This report details an independent 
comparison of these four standards sets using 
the American Diploma Project (ADP; Achieve, 
Inc. 2004) standards set as the benchmark and 
the other three as comparison sets. 

The Commission for a College Ready Texas 
(2007), which was guiding the development of 
college readiness standards, requested technical 
assistance from Regional Educational Labora- 
tory (REL) Southwest for a comparison of Eng- 
lish language arts college readiness definitions 
in the four standards sets. No previous inde- 
pendent comparisons had been identified. Once 
this study was complete, members of the REL 
Southwest Governing Board saw the technical 
assistance as relevant to college readiness stan- 
dards work being conducted in other states in 
the Southwest Region that had not gone through 
a process of internally developing and formally 
adopting their own college readiness standards. 

The board requested that the study be replicated 
using a more rigorous methodology so that the 
results could inform policymakers, curriculum 
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experts, standards-writing and review teams, 
and state assessment writing teams about simi- 
larities and differences in content and cognitive 
complexity between the ADP standards and 
each of the three comparison sets of college 
readiness standards for English language arts: 
the ACT College Readiness Standards (ACT; 
ACT, Inc. 2007), College Board College Readi- 
ness Standards (College Board 2006), and Stan- 
dards for Success (S4S; Conley 2003). 

Building on the initial technical assistance 
work, this two-part study includes a system- 
atic examination of the content of the stan- 
dards statements (the knowledge and skills 
explicitly stated or strongly implied) and an 
analysis of their cognitive complexity (the level 
of reasoning, cognitive demand, or depth of 
knowledge required to demonstrate mastery 
of the contents of a standards statement). ADP 
was again selected as the benchmark because 
the ADP standards set includes statements 
that represent the content deemed necessary 
by college readiness standards experts at a 
level of detail that is easily communicated to 
both policymakers and content experts (not 
too specific or too broad), because 35 states are 
part of the ADP network, and because several 
Texas policymakers were involved in develop- 
ing the ADP standards. While ADP was thus 
considered the most appropriate choice for the 
benchmark in the this study, any standards set 
could have been used as the benchmark, and 
ADP’s selection does not imply superiority. 

The report addresses two primary research 
questions: 

• For what percentage of content statements 
in the American Diploma Project college 
readiness standards set (the benchmark) 



is there a completely or partially aligned 
content statement in each of the other 
three sets of comparison standards (ACT, 
College Board, Standards for Success)? 

• For each standards set what is the dis- 
tribution of content statements across 
the four levels of a cognitive complexity 
(cognitive demand) scale? 

Alignment of the standards statements in 
each of the three comparison sets to the ADP 
standards statements was established by sys- 
tematically comparing individual standards 
statements to determine whether content was 
shared (content alignment) and, if so, at what 
level (using a three-level content alignment 
rating scale — complete, partial, no align- 
ment). The cognitive demand expected of 
students in each college readiness standards 
statement also was rated using Webb’s (2002) 
four-level depth of knowledge (DoK) scale, 
which is typically used to evaluate the cogni- 
tive complexity alignment of test items to 
standards (Rothman 2004). 

Among the study findings, four stand out. 
First, the percentage of ADP’s 62 standards 
statements that align with standards state- 
ments in each of the comparison sets var- 
ies, from 77 percent completely or partially 
aligned statements in College Board to 68 
percent in S4S, and 34 percent in ACT. Second, 
only 5 percent of ADP standards statements (3 
of 62) completely align with content included 
in all three comparison sets. When partial 
alignment is also considered, the content 
shared by all four sets of standards rises to 27 
percent (17 of the 62 ADP statements). Third, 
each set of standards contains content that 
does not align to ADP content — 51 percent of 



ACT statements, 30 percent of College Board 
statements, and 15 percent of S4S statements. 
Fourth, all four levels of the DoK scale are 
represented in each of the college readiness 
standards sets, although more than half the 
statements in each set of standards are written 
at level 3-strategic thinking, which requires 
students to demonstrate reasoning, planning 
skills, and the ability to make complex infer- 
ences. State standards and assessments at cog- 
nitive complexity levels 1 and 2 may therefore 
not reflect the level of demand intended by 
many college readiness standards. 

The study has several limitations. Only one 
set of college readiness standards (ADP) was 
used as the benchmark, so a direct analysis of 
the content alignment between ACT, College 
Board, and S4S was not done. The standards 
sets align only on general content and cogni- 
tive complexity, not on other potentially useful 
dimensions — such as breadth, depth, and 



specificity — that would provide additional 
content detail that state standards writing 
teams or assessment writing teams might 
find useful. No statement can be made about 
the superiority of one set of standards over 
another or about the degree to which mas- 
tery of the skills defined by the standards is 
associated with success in college (with the 
exception of ACT 1 ). In addition, the manner in 
which the terms complete alignment, partial 
alignment, and no alignment were defined and 
interpreted, and the subjectivity inherent in 
assigning ratings (an issue for all alignment 
studies), could have affected the findings. 

February 2010 



1. The link between high ACT scores, first-year 
college success, and specific standards mastery 
has been established (ACT, Inc. 2007). 
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This study of four national 
English language arts college 
readiness standards sets 
compares content alignment 
and level of alignment of the 
standards statements in three 
comparison sets to a benchmark 
set, the American Diploma 
Project (ADP), and analyzes 
the cognitive complexity of all 
four sets. Standards statements 
in the comparison sets align 
completely or partially to 
varying proportions of the 
ADP benchmark's 62 standards 
statements — 77 percent for 
the College Board College 
Readiness Standards, 68 percent 
for Standards for Success, 
and 34 percent for the ACT 
College Readiness Standards. 
But only 5 percent of the 
ADP statements completely 
align with content in all three 
comparison sets, a share that 
rises to 27 percent when partial 
alignment is also considered. 

A majority of statements in the 
four sets (53-68 percent) were 
rated level 3 on a four-level 
cognitive complexity scale. 



WHY THIS STUDY? 

The 1983 publication of A Nation at Risk called 
for “schools, colleges, and universities [to] adopt 
more rigorous and measurable standards, and 
higher expectations for academic performance” 
(National Commission on Excellence in Educa- 
tion 1983 as cited in U.S. Department of Education 
2008, p. 5). Thus began the national movement 
to develop high standards for instruction for all 
students, also known as standards-based reform. 
While the adoption of K-12 standards (statements 
defining the knowledge and skills that students 
should have in specific content domains as they 
progress from kindergarten through grade 12) was 
initially voluntary, it was eventually required by 
federal legislation beginning with the Improving 
America’s Schools Act of 1994 (1995) and followed 
by the No Child Left Behind (NCLB) Act of 2001 
(No Child Left Behind Act 2002). 

While all states have adopted K-12 standards, the 
proper alignment of these standards to the de- 
mands of postsecondary education (often termed 
P-16 alignment) is not federally mandated. State 
efforts in this area lag behind the establishment 
of rigorous K-12 standards (Achieve, Inc. 2008). 
Some states such as Texas, however, have devel- 
oped separate college readiness standards (Texas 
Higher Education Coordinating Board 2008). 
College readiness standards define the knowledge 
and skills thought to be required for students 
to succeed in the first year of a four-year college 
program (ACT, Inc. 2008a). 



Texas initiative to develop college readiness standards 

In 2006 the Texas legislature passed House Bill 1, 
Section 5.01, which called for the development of 
college readiness standards and the formation of 
the Commission for a College Ready Texas (CCRT) 
to guide the effort (Commission for a College Ready 
Texas 2007). The CCRT invited expert testimony 
from four organizations that had developed college 
readiness standards for national use: the American 
Diploma Project (ADP; Achieve, Inc. 2004), the 
ACT College Readiness Standards (ACT, Inc. 2007), 
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AMERICAN DIPLOMA PROJECT ENGLISH LANGUAGE ARTS COLLEGE READINESS STANDARDS 



This study focuses 
on college readiness 
standards for 
English language 
arts and examines 
two dimensions of 
alignment: content and 
cognitive complexity 



College Board College Readiness 
Standards (College Board 2006), 
and Standards for Success (S4S; 
Conley 2003). These organizations 
had consulted with various states 
in developing more rigorous K-12 
standards that encompass college 
readiness standards and require- 
ments (Achieve, Inc. 2008; ACT, 
Inc. 2008b; College Board 2008; 
Conley 2007). Because of the varied nature and 
volume of these college readiness standards, the 
CCRT requested technical assistance from Regional 
Educational Laboratory (REL) Southwest in evalu- 
ating similarities across the four sets of standards 
to ensure that essential knowledge and skills were 
reflected in the Texas standards. 1 



To meet the CCRT’s time constraints, REL South- 
west proposed to align three of the sets to a fourth 
set (designated as the benchmark set of standards) 
using a single-reviewer alignment methodology 
that aligns statements based on shared content as 
defined by one content expert’s opinion. The CCRT 
requested that Achieves ADP standards set be 
used as the benchmark, in part because this set is 
widely used (currently in 35 states; Achieve, Inc. 
2009) and in part because several Texas stake- 
holders participated in the original meetings to 
develop this standards set (Achieve, Inc. 2004). 

The findings of the initial technical assistance study 
(Commission for a College Ready Texas 2007) were 
valuable in the CCRT effort. 2 Once the study was 
complete, members of the REL Southwest Govern- 
ing Board (including all five state education chiefs) 
requested that REL Southwest conduct a study with 
a more rigorous methodology. The Governing Board 
members saw the technical assistance as relevant to 
college readiness standards work being conducted 
in other states in the Southwest Region that had not 
gone through a process of internally developing and 
formally adopting their own state-specific college 
readiness standards. The importance of college 
readiness standards is evidenced by stipulations 
in the American Recovery and Reinvestment Act 
of 2009 that states requesting stimulus funds for 



education show “[progress toward adopting stan- 
dards and assessments that prepare students to suc- 
ceed in college and the workplace and to compete in 
the global economy” (U.S. Department of Education 
2009, para. 3). 



The current study 

The current study, which focuses on college 
readiness standards for English language arts, 3 
examines two dimensions of alignment: content 
and cognitive complexity. While researchers have 
defined other dimensions by which standards can 
be described and aligned, such as breadth, depth, 
and specificity (La Marca 2001; Rothman 2004), 

La Marca (2001, para. 4) concluded that content 
knowledge and cognitive complexity were the “two 
overarching dimensions” of alignment, and Texas 
policymakers and educators identified them as the 
primary alignment dimensions of interest. 4 

This study defines content as the knowledge and 
skills explicitly stated or strongly implied in a 
standards statement (such as “demonstrate knowl- 
edge of 18th and 19th century foundational works 
of American literature and write an academic 
essay”). It defines content alignment as the identi- 
fication of content in a statement (or statements) 
from one set of standards (a comparison set of 
standards) as the same as content in a statement 
from another set of standards (the benchmark set). 

The study also examines the cognitive complex- 
ity of both the individual statements and the 
standards sets as a whole. Cognitive complexity is 
defined as the level of cognitive demand, depth 
of knowledge, or reasoning (level of abstraction, 
number of steps, type of thinking) required to 
demonstrate the knowledge or skills represented 
by a standards statement (Rothman 2004; Webb 
1999). Knowing the level of cognitive complexity 
is useful to ensure that test items in state assess- 
ments are measuring state curriculum standards 
at the appropriate level of difficulty (Nasstrom and 
Henriksson 2008). Knowing the aggregate distri- 
bution of the statements at various levels of cogni- 
tive complexity was hypothesized to be useful 
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for identifying differences in college readiness 
expectations in each of the four sets of standards. 

The current study employs a more rigorous ap- 
proach than the initial technical assistance study 
in four ways: two independent reviewers were used 
instead of one; 5 in addition to content alignment, 
the level of content alignment (completely, partially, 
not aligned) between statements in the comparison 
standards sets and the ADP benchmark statements 
was rated; a consensus process between the two 
reviewers and a third, senior reviewer, was imple- 
mented to finalize decisions on the level of content 
alignment (alignment ratings); and a second dimen- 
sion of alignment was evaluated by assigning cogni- 
tive complexity ratings to the standards statements 
using the same consensus process. 

The same content alignment design was employed 
to determine the level of alignment between ADP 
standards statements and ACT, College Board, 
and S4S in order to build on the previous CCRT 
work (the steps involved in aligning more than one 
comparison set to a benchmark, the determination 
that the benchmark alignment methodology was 
easily understood and well received by policy- 
makers). 6, 7 While any standards set could have 
been employed as the benchmark, using the ADP 
standards set was considered most appropriate for 
this study for several reasons: the ADP standards 
statements of the content deemed necessary by 
college readiness standards experts are presented 
at a level of detail that is easily communicated to 
both policymakers and content experts (not too 
specific or too broad), 35 states are part of the ADP 
network, and several Texas policymakers were 
involved in developing the ADP standards. 



information that may be useful to policymakers 
in their own standards development. For cur- 
riculum experts and members of state college 
readiness standards-writing or review teams, a 
detailed table describing the level of alignment of 
each ADP standards statement with statements in 
comparison standards sets is available from REL 
Southwest to inform their work of examining ex- 
isting standards sets for 
agreement, disagreement, 
and exemplars. For state 
assessment writing teams 
the cognitive complexity 
ratings can inform the 
development and align- 
ment of individual test 
items with individual 
statements in terms of 
the level of cognitive 
demand. 



This report 

communicates the broad 
issues on which there is 
substantial agreement 
and disagreement and 
provides information 
that may be useful 
to policymakers in 
their own standards 
development 



Research questions 

The primary research questions addressed in this 

report are: 

• For what percentage of content statements 
in the American Diploma Project college 
readiness standards set (the benchmark) is 
there a completely or partially aligned content 
statement in each of the other three sets of 
comparison standards (ACT, College Board, 
Standards for Success)? 

• For each standards set what is the distribution 
of content statements across the four levels of a 
cognitive complexity (cognitive demand) scale? 



This report is intended to be used in several ways. 
For policymakers the body of the report contains 
a high-level content comparison of college readi- 
ness standards sets using the ADP standards as 
the benchmark and the distribution of statements 
from each standards set across four levels of 
cognitive complexity. This information communi- 
cates the broad issues on which there is substan- 
tial agreement and disagreement and provides 



COLLEGE READINESS STANDARDS SETS AND 
COGNITIVE COMPLEXITY FRAMEWORK 

This section details the four sets of English lan- 
guage arts college readiness standards used in this 
study (summarized in table 1) — describing the 
development processes, goals of the developing or- 
ganizations, intended uses, and strand structures 
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TABLE 1 

Overview of the four sets of college readiness standards and their English language arts strands, 2008 





American Diploma 








Item 


Project 


ACT 


College Board 


Standards for Success 


Year first published 


2004 


2007 


2006 


2003 


Publisher 


Achieve, Inc. 


ACT, Inc. 


College Board 


University of Oregon 
Center for Educational 
Policy Research 


Organization type 


Education reform 


Test publisher 


Test publisher 


University researcher 




organization 






in partnership with 




promoting 






Pew Charitable Trust 




postsecondary 






and the American 




readiness 






Association of 
Universities 


Method for deriving 


Committees of 


National Curriculum 


Expert standards 


Committees of 


standards statements 


postsecondary 


Survey to inform 


advisory committee of 


postsecondary faculty 




academic leaders and 


test development — 


selected high school 


and representatives 




business leaders 


standards derived from 


and postsecondary 


from 40 prominent 






test content 


academic leaders 


universities 


English language arts 


• Communication 


• English 


• Listening 


• Critical thinking 


strands 


• Informational text 


• Reading 


• Media literacy 


skills 




• Language 

• Literature 

• Logic 

• Media 

• Research 

• Writing 


• Writing 


• Reading 

• Speaking 


• Reading and 
comprehension 


• Research skills 

• Writing 


• Writing 












Number of English 
language arts 
standards statements 


62 


191 


115 


73 



Source: Achieve, Inc. 2004; ACT, Inc. 2007; College Board 2006; Conley 2003. 



of each set — and explains the framework used 
for characterizing the cognitive complexity of 
standards. 



Description of college readiness standards sets 

The English language arts domain of each set 
of college readiness standards is organized into 
strands, or clusters of related standards state- 
ments. 8 For example, the College Board speaking 
strand contains the individual statements “Under- 
stands how speakers’ and listeners’ internal vari- 
ables affect communication” and “Understands 
how contextual variables affect communication” 
(Sl.1.2 and SI. 1.3; College Board 2006). Strand 
names vary across the standards sets, and the 



organization of statements into strands can help 
identify areas of emphasis. 

American Diploma Project. The ADP, created by 
Achieve, Inc., has assembled a network of state 
policymakers and other leaders to align state stan- 
dards and assessments and raise them to a level 
that will prepare students for success in postsec- 
ondary education. As of 2009, 35 states were part 
of the ADP network (Achieve, Inc. 2009). 

The ADP standards were developed through a 
two-year process that solicited input from busi- 
ness leaders and postsecondary educators from five 
states, including Texas (the others were Indiana, 
Kentucky, Massachusetts, and Nevada; Achieve, 
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Inc. 2004). This group identified prerequisite 
knowledge and skills for success in postsecondary 
education such as entry-level English courses. A 
working set of standards representing content in 
the domains of English and math emerged from 
this research as a basis for refining state K-12 stan- 
dards and assessments. The ADP English language 
arts standards are divided into eight strands: com- 
munication, informational text, language, litera- 
ture, logic, media, research, and writing. 

ACT. The ACT College Readiness Standards, de- 
veloped by ACT, Inc., are intended to represent the 
range of knowledge and skills that most students 
should be able to demonstrate based on their 
scores on the ACT assessments (ACT, Inc. 2007). 
Students receive individual results, and their 
performance relative to the standards is intended 
to assist students, parents, and teachers in identi- 
fying individual skill deficits and assist teachers in 
modifying instruction to address student needs. 

The ACT assessment standards were developed 
through a multistage process by ACT, Inc. staff 
and reviewed by scholars (identified by ACT as 
nationally recognized) from high school and 
university English and reading education depart- 
ments. Based on the distribution of student scores 
on ACT’s Educational and Planning Assessment 
System and 40 years of research on ACT student 
assessment data, ACT identified eight score 
ranges that most accurately identified students’ 
levels of achievement. Four ACT content teams 
reviewed several forms of the ACT assessments 
by content domain — English, math, science, and 
reading — and conceptualized what each ACT 
assessment measured. ACT staff wrote the college 
readiness standards based on their expert analy- 
sis of the knowledge and skills a student needs to 
respond correctly to the assessment items. Finally, 
independent reviewers validated the English 
language arts college readiness standards, which 
were divided into three strands: English, reading, 
and writing. 

College Board. The College Board Standards for 
College Success were designed to increase the 



success of students enrolled in first-year college 
courses, to increase their scores on the SAT, to in- 
crease college attendance and college completion, 
and to reduce college remediation rates. College 
Board standards were developed in two content 
domains — English language arts, and math and 
statistics — to provide a framework of model 
courses for states and districts to follow in prepar- 
ing students for college (College Board 2006). 

The Expert Standards Advisory Committee — 
composed of postsecondary teacher education 
faculty, middle and high school teachers, and 
assessment and curriculum specialists with ex- 
perience in developing standards — developed the 
standards over four years 
using a multistep expert 
judgment process. The 
committee first identi- 
fied the English language 
arts knowledge and skills 
required for entry-level 
college students. Then 
working backward from 
these skills, the com- 
mittee identified the 
prerequisite knowledge 
and skills from grade 6 
through college. These 
skill sets subsequently 
became sets of standards. 

The College Board set of 
standards for English language arts define per- 
formance expectations for five strands: listening, 
media literacy, reading, speaking, and writing. 

Standards for Success. The S4S set of standards 
was developed by Dr. David Conley at the Uni- 
versity of Oregon Center for Educational Policy 
Research under a grant from the Pew Charitable 
Trusts in partnership with the American Associa- 
tion of Universities (Conley 2003, 2005). The S4S 
require students to correctly use and apply gen- 
eral concepts to interpret or explain more specific 
knowledge and skills. The standards represent six 
content domains: English, math, natural sciences, 
social sciences, second languages, and the arts. 



The English language 
arts domain of each set 
of college readiness 
standards is organized 
into strands, or clusters 
of related standards 
statements. Strand 
names vary across the 
standards sets, and 
the organization of 
statements into strands 
can help identify 
areas of emphasis 
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BOX 1 

Study methodology and ratings 
scales 

This box describes the methodology 
and rating scales used to examine con- 
tent alignment and cognitive complex- 
ity (for more detail, see appendix A). 

Content alignment methodology. Con- 
tent alignment is the identification 
of content in a statement from one 
set of standards (a comparison set 
of standards) as the same as content 
in a statement from another set (the 
benchmark set). The content align- 
ment rating indicates the level of con- 
tent alignment on a three-level scale 
(complete, partial, no alignment). 

This study adapted the content align- 
ment methodology used in a previ- 
ous series of REL Southwest studies 
(Timms et al. 2007; Shapley and Brite 
2008), employing the same three-level 
content alignment scale and the same 
process for reconciling independent 
ratings. It follows the same pair-wise 
comparison approach, individually 
aligning the 191 standard statements 
of the ACT, the 115 statements of 
the College Board, and the 73 state- 
ments of the S4S to the 62 standard 
statements of the ADR Three content 
alignment tables were created to 
conduct these pair-wise comparisons, 
with the first column populated with 
ADP standards statements. Two raters 
used the following three-level content 
alignment scale to rate the level of 
content alignment at the statement 
level (see appendix A for details): 

• Complete alignment. All content 
in the benchmark statement 



aligns with content in the com- 
parison standards set. 

• Partial alignment. Some of the 
content (1-99 percent) in the 
benchmark statement aligns 
with some portion of the content 
in the comparison standards set. 

• No alignment. None of the con- 
tent in the benchmark statement 
aligns with any of the content in 
the comparison standards set. 

Final alignments and ratings were de- 
termined during a consensus meeting 
with a third senior reviewer. An ex- 
ample of how each content alignment 
table was structured and populated is 
provided in figure A2 in appendix A. 

Although the two reviewers inde- 
pendently aligned the standards sets 
using the three content alignment 
tables, for ease of reference and 
greater utility the results for each 
pair-wise comparison are repre- 
sented in a single alignment table 
(available upon request) instead of 
as separate tables for each pair. The 
findings are also presented by strand 
in appendix C. Only statements from 
the comparison standards sets that 
could be aligned to ADP statements 
appear in the alignment tables; the 
statements that could not be aligned 
are provided in appendix D. 

Cognitive complexity rating methodol- 
ogy. The cognitive complexity rating 
indicates the depth of knowledge re- 
quired to demonstrate mastery of the 
knowledge and skills represented by a 
standards statement. Cognitive com- 
plexity was assessed by two reviewers 



who independently compared the dis- 
tribution of standards statements from 
each set of standards across four levels 
of cognitive complexity using Webb’s 
(2002) depth of knowledge (DoK) scale 
(see appendix E for details): 

• Level 1-recall. Requires students 
to use simple skills or abilities to 
retrieve or recite facts. 

• Level 2-skill/concept. Requires 
a level of comprehension and 
subsequent processing across 
portions of text to make infer- 
ences beyond simple recall or 
recitation of stated facts. 

• Level 3-strategic thinking. 
Focuses on reasoning, planning 
skills, making more complex 
inferences, and applying ideas 
from the text; students may be 
encouraged to explain, general- 
ize, or connect ideas. 

• Level 4-extended thinking. Re- 
quires investigation and higher 
order thinking skills to process 
multiple solutions to a given 
problem. 

A two-column cognitive complex- 
ity rating table was created for each 
standards set, with each standards 
statement in the first column and 
the cognitive complexity level in the 
second column. The cognitive com- 
plexity ratings of the two indepen- 
dent reviewers were discussed, and 
final ratings were determined during 
meetings to achieve consensus with 
a senior reviewer. An example of a 
cognitive complexity rating table is 
provided in figure A3 in appendix A. 



(CONTINUED) 
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Review process. The review process 

consisted of eight steps: 

• Step 1-selecting reviewers. Eng- 
lish language arts teachers with 
experience in alignment stud- 
ies were recruited as primary 
reviewers, and an experienced 
researcher was selected as the 
supervising senior reviewer (for 
more information about reviewer 
qualifications, see appendix B). 

• Step 2-training reviewers. The 
senior reviewer conducted a 
three-hour training session for 
the reviewers on the three-level 
content alignment rating scale 
and the Webb (2002) rating scale. 
The primary reviewers then 
independently practiced align- 
ing and rating a small number 
of standards statements, which 
they then discussed with the 
senior reviewer and resolved any 
discrepancies. 

• Step 3 -rating ADP cognitive 
complexity levels. To familiarize 
reviewers with each standards 
statement before content align- 
ment began, reviewers individu- 
ally rated each ADP standards 
statement on the cognitive com- 
plexity scale (see appendix E). 

• Step 4-achieving consensus on 
ADP cognitive complexity levels. 
The two independent reviewers 



met with the senior reviewer 
to compare ADP cognitive 
complexity ratings and reach 
consensus in cases of disagree- 
ment. Reviewers then again 
independently rated 5 percent of 
the statements and compared the 
results with their original ratings 
to check for rater drift. 1 

• Step 5-rating and achieving con- 
sensus on comparison sets’ cogni- 
tive complexity levels. Reviewers 
individually rated each standards 
statement of the comparison sets 
for cognitive complexity (start- 
ing with ACT and moving on to 
College Board and finally S4S) 
and then met with the senior 
reviewer to compare ratings and 
achieve consensus. Reviewers 
then again independently rated 

5 percent of the statements and 
reviewed them for rater drift. 

• Step 6-comparison and align- 
ment of ADP-ACT content. 

Using the ADP-ACT content 
alignment table, each reviewer 
independently searched all ACT 
statements for content aligned to 
the ADP benchmark statements. 
Once all completely and partially 
aligned ACT statements were 
identified, the reviewer assigned 
a content alignment rating based 
on the cumulative content of 

all the aligned ACT statements 
to the ADP standard statement 



(complete alignment, partial 
alignment, or no alignment). 
Meetings to achieve consensus 
were held after completion of 
every two ADP strands until all 
ADP statements were aligned 
and content alignment levels 
were rated. Reviewers then again 
independently rated 5 percent of 
the ADP statements and re- 
viewed them for rater drift. 

• Steps 7 and 8-comparison and 
alignment of ADP- College Board 
and ADP-S4S content. The same 
process as in step 6 was followed 
for ADP-College Board and 
ADP-S4S content alignment. 

This study can be seen as three sepa- 
rate content alignment studies. The 
methodology (pair-wise comparison 
of three sets to a single benchmark 
set) is consistent with the initial work 
conducted for the Commission for a 
College Ready Texas, but it is limited 
in several ways (see section in report 
on limitations and suggestions for 
further research). 

Note 

1. Rater drift is the tendency for raters or 

assessors to unintentionally redefine cri- 
teria over time. Because drift occurred 
so infrequently (zero to one occurrence 
per weekly check), instances were not 
formally recorded, and the drift that did 
occur did not influence the final consen- 
sus ratings for either content alignment 
or cognitive complexity ratings. 



A group of 400 faculty members representing 20 
universities participated in meetings to identify 
a broad range of skills that students should pos- 
sess to perform well in entry-level postsecondary 
courses. Neither the universities nor partici- 
pants were selected to be representative of such 



institutions as a whole, but they did cover a range 
of institutional sizes and geographic diversity. 

The S4S English language arts statements are 
divided into four strands: critical thinking skills, 
reading and comprehension, research skills, and 
writing. 
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Description of cognitive complexity framework FINDINGS 



In addition to the content specified in a standard, 
stakeholders interested in creating or modifying 
curriculum standards for college readiness may 
need to attend to how students are expected to 
manipulate or express knowledge and skills. Stan- 
dards statements can communicate the difficulty 
level, or demand, intended through the use of 
specific language and key terms (Rothman 2004; 
Webb 1997, 1999, 2002). The demand embodied 
in a statement can strongly influence the develop- 
ment of instructional materials and assessments. 
For example, statements that require students only 
to “identify” or “recognize” certain content would 
require lower levels of knowledge and skills than 
standards that require students to “reason with,” 
“synthesize,” or “produce” complex materials. 



Only 5 percent of ADP 
statements completely 
align with all three 
comparison sets of 
standards. That share 
rises to 27 percent if both 
partial and complete 
alignment are considered 



For this study, the Webb (2002) depth of knowl- 
edge (DoK) scale was selected for evaluating dif- 
ferences and similarities in the cognitive demand 
required by each of the college readiness standards 
sets (see box 1 and appendix A for details). Using a 
four-level DoK scale (recall, skill/concept, strate- 
gic thinking, and extended thinking) to examine 
standards statements in four states, Webb (1999) 
found that DoK ratings varied substantially across 
statements representing the same content and that 
the distribution of ratings across the four levels 
differed by state. Thus the DoK scale appeared 

to be a useful differentiator for 

understanding the level of demand 
expressed by different state docu- 
ments. The College Board used 
the DoK scale to assess the level of 
cognitive demand expected when 
describing the alignment between 
expectations for student learning 
articulated in Texas K-12 stan- 
dards (the Texas Essential Knowl- 
edge and Skills) and the SAT 
(College Board 2005). The DoK scale has also been 
used in other studies (Webb 1997, 2002; Wixson 
et al. 2002) to assess depth of knowledge and was 
therefore adopted to measure cognitive complexity 
in the current study. 



The level of interrater agreement can provide an 
important context for interpreting study results, so 
it is discussed before the results on content align- 
ment and cognitive complexity. 



Interrater agreement 

In general, high levels of agreement in studies 
employing expert judgments suggest that the 
rating scales, reviewer training, and alignment 
methodology were appropriate and that the 
findings are replicable. High levels of interrater 
agreement are especially important in studies that 
compute a mean rating from several raters (for 
example, Webb, Herman, and Webb 2007). This 
study did not compute a mean rating but used 
a consensus-forming process to determine the 
final ratings. Interrater results are described here 
to provide context for the interim rating process 
(before consensus). The level of agreement in this 
interim rating process is acceptable given the 
consensus process that followed (the procedures 
for calculating two interrater agreement measures 
are discussed in appendix B). 

Interrater agreement for subjective judgments is 
rarely perfect. Results should be interpreted relative 
to agreement levels found in similar studies. All in- 
terrater agreement measures were calculated using 
individual ratings prior to the consensus process. 
While the senior reviewer was to make the final 
decision in cases where the independent reviewers 
could not reach consensus, this process never had to 
be invoked to resolve discrepancies in this study. 

The two independent reviewers achieved 73 per- 
cent agreement with an intraclass correlation of 
0.78 for the ADP-ACT content alignment ratings, 
a 48 percent agreement rate with an intraclass 
correlation of 0.69 for the ADP-College Board 
content alignment, and a 69 percent agreement 
rate with an intraclass correlation of 0.57 for the 
ADP-S4S content alignment. For comparison, 
only one recent study of curriculum to standards 
alignment was identified that also reported 
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interrater reliability (Porter et al. 2008). Porter 
et al. reviewed English language arts alignment 
studies in two states at three grades. They calcu- 
lated G-coefhcients (equivalent to the intraclass 
correlations reported here — see tables B1 and B2 
in appendix B) of 0.47-0.83 for two raters. The 
intraclass correlation in the current study are 
within the same range; however, the alignment 
methodologies are not directly comparable. 

The two independent reviewers achieved a 75 per- 
cent agreement rate with an intraclass correlation of 
0.77 for the ADP cognitive complexity ratings, a 46 
percent agreement rate with an intraclass correlation 
of 0.67 for the ACT cognitive complexity ratings, a 
54 percent agreement rate with an intraclass correla- 
tion of 0.50 for the College Board, and a 53 percent 
agreement rate with an intraclass correlation of 0.62 
for the S4S. These findings are within the broad 
range found in Webb, Horton, and O’Neal (2002, p. 
11) who report intraclass correlations of 0.36-0.92 
(M = 0.73) for cognitive complexity ratings of Eng- 
lish language arts assessment items. The results of 
the current study are not directly comparable to the 
results of Webb, Horton, and O’Neal (2002) because 
of differences in what was being rated (test items 
in Webb, Horton, and O’Neal and standards in the 
current study). In addition, since the final ratings in 
the current study were determined using a consen- 
sus methodology, the degree of initial agreement is 
not critical to the final consensus ratings for content 
alignment or cognitive complexity. 



Content alignment findings 

Alignment to ADP standards statements. A 

primary goal of this study was to determine the 
percentage of agreement between the skills and 
knowledge ADP identifies as essential for college 
readiness and the skills and knowledge each of 
the three comparison sets of college readiness 
standards identifies as essential. Alignment tables 
C1-C8 in appendix C were examined for ADP 
content also contained in the other standards sets. 
Complete alignment was stringently defined for 
this study. Only 5 percent of ADP statements (3 of 
62) completely align with all three comparison sets 



of standards (bolded rows in table 2). That share 
rises to 27 percent (17 of 62) if both partial and 
complete alignment are considered (table 2). 



At the broadest level each of the three pair-wise 
comparisons can be characterized by the percent- 
age of content statements in the ADP standards 
set (the benchmark) that completely or partially 
align with content in the 
comparison standards 
set (ACT, College Board, 
and S4S). These results 
identify knowledge and 
skills that are considered 
important for English 
language arts college 
readiness by ADP and at 
least one other set of col- 
lege readiness standards. 



The levels of agreement 
with ADP among the 
comparison sets varies 
considerably. The 
share of ADP standards 
statements with 



complete or partial 
alignment is 34 percent 
for ACT standards 
statements, 77 percent 
for College Board 
standards statements, 
and 68 percent for S4S 
standards statements 



The levels of agreement 
with ADP among the 
comparison sets varies 
considerably (figure 1). 

The share of ADP stan- 
dards statements with complete or partial align- 
ment is 34 percent (21 of 62 standards statements) 
for ACT standards statements, 77 percent (48 of 
62) for College Board standards statements, and 68 
percent (42 of 62) for S4S standards statements. 



Alignment to ADP strands Figure 2 summarizes 
the percentage of ADP standards statements 
within each of the eight strands that align at each 
level (complete, partial, no alignment) with the 
comparison sets. Statements in the ADP infor- 
mational text, writing, and language strands 
completely or partially align with ACT at levels 
of 50-71 percent. The ADP literature and logic 
strands statements are minimally addressed by 
ACT, and the ADP media, research, and communi- 
cation strands contain content that does not align 
with any ACT statements. 

Standards statements in all eight ADP strands 
are completely or partially aligned with Col- 
lege Board statements at levels of approximately 
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TABLE 2 

The 17 (of 62) standards statements of the American Diploma Project for which comparison sets exhibited 
complete or partial alignment in 2008 (number of statements aligned) 



American Diploma Project strands and statements 


ACT 


College 

Board 


Standards 
for Success 


Language 


A1. Demonstrate control of standard English through the use of grammar, punctuation, 
capitalization and spelling. 


28 


2 


10 


A4. Use context to determine the meaning of unfamiliar words. 


1 


1 


2 


A5. Identify the meaning of common idioms, as well as literary, classical and biblical allusions; 
use them in oral and written communication. 


4 


3 


2 


A6. Recognize nuances in the meanings of words; choose words precisely to enhance 
communication. 


3 


5 


2 


Writing 


C2. Select and use formal, informal, literary or technical language appropriate for the purpose, 
audience and context of the communication. 


3 


3 


6 


C3. Organize ideas in writing with a thesis statement in the introduction, well constructed 
paragraphs, a conclusion and transition sentences that connect paragraphs into a coherent whole. 


11 


6 


3 


C4. Drawing on readers' comments on working drafts, revise documents to develop or 
support ideas more clearly, address potential objections, ensure effective transitions between 
paragraphs and correct errors in logic. 


20 


2 


3 


C5. Edit both one's own and others' work for grammar, style and tone appropriate to audience, 
purpose and context. 


33 


4 


2 


C9. Write an academic essay (for example, a summary, an explanation, a description, a literary 
analysis essay) that; develops a thesis; creates an organizing structure appropriate to purpose, 
audience, and context; includes relevant information and excludes extraneous information; 
makes valid inferences; supports judgments with relevant and substantial evidence and well- 
chosen details; and provides a coherent conclusion. 


15 


22 


7 


Logic 


E4. Evaluate the range and quality of evidence used to support or oppose an argument. 


10 


3 


5 


E9. Construct arguments (both orally and in writing) that: develop a thesis that demonstrates 
clear and knowledgeable judgment; structure ideas in a sustained and logical fashion; use a 
range of strategies to elaborate and persuade, such as descriptions, anecdotes, case studies, 
analogies and illustrations; clarify and defend positions with precise and relevant evidence, 
including facts, expert opinions, quotations and/or expressions of commonly accepted beliefs 
and logical reasoning; anticipate and address the reader's concerns and counterclaims; and 
provide clear and effective conclusions. 


5 


38 


9 


Informational Text 


F3. Summarize informational and technical texts and explain the visual components that 
support them. 


1 


6 


2 


F6. Identify interrelationships between and among ideas and concepts within a text, 
such as cause-and-effect relationships. 


17 


2 


2 


F8. Draw conclusions based on evidence from informational and technical texts. 


4 


1 


1 


F9. Analyze the ways in which a text's organizational structure supports or confounds its 
meaning or purpose. 


3 


1 


4 


Literature 


H4. Analyze the setting, plot, theme, characterization and narration of classic and 
contemporary short stories and novels. 


7 


5 


4 


H8. Analyze the moral dilemmas in works of literature, as revealed by characters' motivation 
and behavior. 


6 


1 


1 



Note: The darker shade represents complete alignment to the ADP statement and the lighter shade represents partial alignment. Project strands and state- 
ments in bold are those for which all three comparison standards sets completely align to the ADP. Statement identifier codes, such as A1, were used in the 
study to identify specific standards statements. The codes follow ADP's prescribed coding format; for example, "A" indicates a statement in the language 
strand, and "1” indicates the first standard statement in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); and Conley (2003); see appendixes A-C for details. 
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FIGURE 1 

Percentage of American Diploma Project 
standards statements that completely or partially 
align with ACT, College Board, and Standards for 
Success standards statements, 2008 



Percent 




Comparison standards sets 

Note: The percentages are the sum of the results in tables Cl -C8 in ap- 
pendix C divided bythe total number of ADP statements. 

Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. 
(2007); College Board (2006); and Conley (2003). 



57 percent or greater; the majority of these are 
partial alignments. For the ADP logic strand all 
alignments are partial, and for the ADP research 
strand the College Board standards set completely 
or partially aligns to 100 percent of the state- 
ments. The exception is the ADP language strand, 
in which the 57 percent of ADP statements that 
align with College Board statements are all com- 
plete alignments. 

The entire ADP language strand completely or 
partially aligns with the S4S standards set, and 
64-90 percent of statements in the ADP research, 
literature, informational text, logic, and writing 
strands completely or partially align with S4S. 
However, none of the standards statements in the 
ADP media and communication strands aligns 
with any of the S4S statements. 

ADP’s media and communication strands merit 
attention because only the College Board state- 
ments align to them completely or partially, but 
they do so at high levels of 75 percent (media) and 
86 percent (communication). 



FIGURE 2 

Percentage of American Diploma Project statements by strand at each level of alignment with the three 
comparison sets, 2008 



Comparison standards sets 


Across all strands 


ACT 

College Board 
Standards for Success 


Language 


ACT 

College Board 
Standards for Success 


Communication 


ACT 

College Board 
Standards for Success 


Writing 


ACT 

College Board 
Standards for Success 


Research 


ACT 

College Board 
Standards for Success 


Logic 


ACT 

College Board 
Standards for Success 


Informational text 


ACT 

College Board 
Standards for Success 


Media 


ACT 

College Board 
Standards for Success 


Literature 


ACT 

College Board 
Standards for Success 



■ Complete alignment Partial alignment No alignment 
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50 



75 



100 



Percentage of American Diploma Project statements at each level of alignment 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); and Conley (2003). 
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Statements that do not align. For six ADP state- 
ments in multiple strands, none of the comparison 
standards statements aligns (listed in table D1 in 
appendix D). These statements can be considered 
unique content statements among the four sets of 
standards. It is also important to identify which 
statements from each of the comparison sets could 
not be aligned to ADP, since they represent English 
language arts content that ADP has not defined as 
critical for college and workforce readiness. 

The percentage of statements from each comparison 
set that do not align to an ADP statement was calcu- 
lated as the number of statements that do not align 
to ADP divided by the total number of statements 
in the comparison set (figure 3). Fifty-one percent 
of ACT statements (97 of 191) could not be aligned 
to ADP, and these statements are distributed across 
all ACT strands. Thirty percent of College Board 
statements (35 of 115) could not be aligned to ADP, 
and the majority (25) were in the reading and listen- 
ing strands. Fifteen percent of S4S statements (11 of 
73) could not be aligned to ADP, and the majority 
(6) were in the reading and comprehension strand. 
(The standards statements that could not be aligned 
to ADP are listed by comparison standards set in 
tables D2-D4 in appendix D.) 



Cognitive complexity findings 

To answer the second research question on the 
distribution of standards statements across cogni- 
tive complexity levels within each of the four 
standards sets, each statement within each college 
readiness standards sets was rated. There was no 
benchmark for this evaluation. All statements 
from all sets were rated using the Webb (2002) 
DoK scale (see appendix E). 

More than half the statements in each standards 
set were rated level 3-strategic thinking, which 
emphasizes reasoning, planning, and integration 
of ideas (figure 4). College Board has the highest 
proportion of level 3-strategic thinking ratings (68 
percent), while ADP has the highest proportion of 
level 2-skill/concept ratings (31 percent), ACT and 
S4S have the highest proportion of level 1-recall 



FIGURE 3 

Percentage of standards statements in each 
comparison set that do not align to American 
Diploma Project standards statements, 2008 

Percent 




Comparison standards sets 

Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. 
(2007); College Board (2006); and Conley (2003). 



FIGURE 4 

Distribution of cognitive complexity level ratings 
across the four levels of the Webb depth of 
knowledge scale, by college readiness standards 
set, 2008 (percent) 



Percent 

100 



75 



50 



■ Level 1 -recall 

■ Level 2-skill/concept 
Level 3-strategic thinking 

■ Level 4-extended thinking 



68 



55 
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31 
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American ACT 

Diploma Project 
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College Standards 

Board for Success 



Comparison standards sets 



Note: Components may not sum to 100 percent because of rounding. 

Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in Achieve, Inc. (2004); ACT, 

Inc. (2007); College Board (2006); and Conley (2003) and the depth of 
knowledge scale from Webb (2002). 
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ratings (18 percent and 14 percent), and ADP and 
S4S have the highest proportion of level 4-extended 
thinking ratings (13 percent and 12 percent). The 
S4S statements are the most evenly distributed, 
with at least 12 percent of statements in each level. 
ACT has the smallest proportion of statements at 
level 4-extended thinking. Figures FI and F2 in ap- 
pendix F summarize the distribution of DoK ratings 
for each strand for each college readiness standards 
set. A table with the cognitive complexity ratings 
for each statement in each set of standards is avail- 
able on request from REL Southwest. This detailed 
table is not included with the report for reasons of 
space, but it may help in understanding the level of 
demand implied by statements of particular interest 
to individual readers. 



CONCLUSIONS 

Several findings emerged from this study. First, 
agreement (complete or partial alignment) on the 
content defined as essential for college readiness 
between ADP and the comparison standards sets 
varies from 34 percent to 77 percent of ADP’s 62 
standards statements — 34 percent for ACT, 68 
percent for S4S, and 77 percent for College Board. 
While there is substantial overlap between ADP 
and each of the three comparison sets using a par- 
tial alignment criterion, the definition of college 
readiness clearly differs. 

Second, content identified by all four sets of 
standards as essential for college readiness is very 
limited. Only 5 percent of ADP standards state- 
ments (3 of 62) completely align with all three 
comparison sets, and only 27 percent of ADP stan- 
dards statements (17 of 62) completely or partially 
align with all three comparison sets. Again, this 
finding reveals a lack of agreement on definitions 
of English language arts college readiness among 
the four standards sets. 

Third, each comparison set of standards contains 
content that does not align to ADP content — 51 
percent of ACT statements (97 of 191), 30 percent 
of College Board statements (35 of 115), and 15 



percent of S4S statements 
(11 of 73). Of the compari- 
son sets, S4S has the few- 
est standards statements 
that could not be aligned 
to ADP statements, while 
more than half of ACT’s 
statements could not be 
aligned to ADP bench- 
mark statements. In 

addition, 10 percent (6 of 62) of ADP’s statements 
contain content that does not align with any of the 
three comparison sets of standards. 



While there is substantial 
overlap between ADP 
and each of the three 
comparison sets using 
a partial alignment 
criterion, the definition 
of college readiness 
clearly differs 



Fourth, in all four college readiness standards sets, 
statements were identified at all four levels of cogni- 
tive complexity using Webb’s (2002) four-level DoK 
scale. Flowever, more than half the statements in 
each set of standards are written at level 3-strategic 
thinking, which requires students to demonstrate 
reasoning, planning skills, and the ability to make 
complex inferences. State standards and assess- 
ments requiring lower levels of cognitive complex- 
ity may therefore not capture the level of demand 
intended by many college readiness standards. 

This study reveals substantial differences among the 
four English language arts college readiness defini- 
tions reviewed here. For pair-wise comparisons 
using ADP as the benchmark, there is only partial 
agreement on the knowledge and skills defined by 
ADP, ACT, College Board, and S4S as necessary for 
college readiness in English language arts. While 
the ADP standards alignment with ACT standards 
appears to be distinctly different from alignment 
with the other two standards sets (see figures 1-3 
and tables C1-C8 in appendix C), dropping ACT 
from the comparison sets would raise the propor- 
tion of ADP statements in complete alignment with 
the two remaining standards sets (College Board 
and S4S) from 5 percent to 13 percent (8 of 62 ADP 
statements) and complete or partial alignment from 
27 percent to 55 percent (34 of 62 ADP statements). 

The key finding for policymakers is the variability 
in how well the three comparison college readiness 
standards sets align to the ADP standards set. The 
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empirical research literature has not evaluated sets 
of college readiness standards and offers no evi- 
dence that one set of standards would lead to higher 
student achievement than another. Thus, it is left to 
state policymakers and experts to make informed 
decisions about what content most closely reflects 
college readiness. Using only one of these four sets 
to inform the development of state college readi- 
ness standards and assessments risks overlooking 
content that should be considered for inclusion. 



LIMITATIONS AND SUGGESTIONS 
FOR FURTHER RESEARCH 



An important limitation of this study is the use of 
a single benchmark (ADP) to examine the four sets 
of standards. That methodology allows observa- 
tions to be made only through the framework of 
ADP. Any of the four sets could have been employed 
as the benchmark, and ADP was selected based 
on regional factors. While the methodology was 
appropriate for the purposes of the current study 
(examining the similarities and differences in the 
content of the three comparison sets as aligned to 
ADP content), it does not allow direct analysis of 
the alignment between the content contained in the 
comparison standards sets that is not included in 
the ADP standards (for example, content shared by 
ACT and College Board that is not in ADP). 



Another major limitation resulted from the study 
methodology that compared the standards only on 
general content and cognitive complexity. While the 



An important limitation 
of this study is that 
the use of a single 
benchmark does not 
allow direct analysis of 
the alignment between 
the content contained 
in the comparison 
standards sets that 
is not included in the 
ADP standards 



findings can be used at a broad 
level to guide policymakers as 
they develop strategies for imple- 
menting P-16 standards align- 
ment, the findings would not be 
as informative for state standards 
writing teams or assessment 
writing teams developing college 
readiness standards or test items 
at a level that includes additional 
useful content dimensions (for 
example, breath, depth, and 
specificity). 



A third limitation of this study is that no statement 
can be made about the superiority of one set of 
standards over another. Only ACT, Inc. (2007) has 
provided predictive validity evidence that estab- 
lishes a clear link between performance on the 
ACT items that are linked to specific standards and 
first-year college course performance. This type of 
link does not exist (at least not in published form) 
for the other three college readiness standards sets. 

The way the three-level content alignment scale 
(complete alignment, partial alignment, and no 
alignment) was defined and interpreted is also a 
limitation. For example, both 90 percent alignment 
and 10 percent alignment qualified as partial align- 
ment. Modifications to these rating definitions 
could lead to different results across the standards 
sets, and the subjectivity inherent in assigning 
these ratings could affect the levels at which state- 
ments align. 9 Future studies might modify these 
definitions of the ratings, for example, using a con- 
tent alignment scale with more than three levels 
and with multiple partial alignment levels (such as 
more than half and less than half). 

Another logical extension of the study for other 
audiences would be to use each of the four sets in 
turn as a benchmark, but it would be difficult to 
integrate findings across four benchmarks. An 
alternative approach would use a set of external 
benchmark statements, as in Kendall et al. (2007), 
who derived a list of topics from a database of 
standards statements in a specific content domain. 
Until such a benchmark set is developed and 
validated as representative of college readiness 
content, its use may be just as arbitrary (or more 
so) as use of any of the four established national 
college readiness standards sets as benchmarks. 

Future studies could also use more than two review- 
ers. Doing so might increase reliability and gener- 
alizability (Webb, Herman, and Webb 2007, p. 25). 

In the current study the two reviewers were reading 
specialists; the addition of more reviewers would 
allow the use of experts with extensive knowledge in 
other English language arts strands, which could re- 
sult in more accurate and reliable content matching. 
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APPENDIX A 
METHODOLOGY 

This appendix describes the methodology and 
rating scales used to examine content alignment 
and cognitive complexity. Content alignment is 
defined as the identification of content in a state- 
ment (or statements) from one set of standards 
(a comparison set of standards) that is the same 
as content in a statement from another set of 
standards (the benchmark set of standards). 
Cognitive complexity is defined as the depth 
of knowledge required for a student to demon- 
strate the knowledge and skills represented by 
a standards statement. The content alignment 
and cognitive complexity ratings were done 
independently. 



Content alignment methodology 



Standards (College Board 2006), and Standards 
for Success (S4S; Conley 2003)— were individu- 
ally aligned to the benchmark standards state- 
ments (figure Al). 

Content alignment scale. Three content alignment 
tables (later combined into one) were created to 
conduct the pair-wise comparisons (ADP-ACT, 
ADP-College Board, ADP-S4S). In each table 
the leftmost column was populated with ADP 
standards statements. The content alignment was 
conducted at the statement level by two indepen- 
dent reviewers using a three-level content align- 
ment scale: 

• Complete alignment. All the content in the 
benchmark (ADP) standards statement aligns 
with content in the comparison standards set 
(ACT, College Board, or S4S). 



The content alignment methodology used in a pre- 
vious series of Regional Educational Taboratory 
(REL) Southwest studies (Timms et al. 2007; Shap- 
ley and Brite 2008) was adapted for the current 
study. The previous studies involved the content 
alignment of two sets of assessment standards to 
the National Assessment of Educational Progress 
(NAEP) assessment standards (the benchmark). 10 
The current study uses the same three-level 
content alignment scale and the same process for 
reconciling independent ratings. Codes represent- 
ing higher and lower grade alignment, employed 
in the NAEP studies, were not used in this study 
because such codes are not relevant for college 
readiness standards, which have only one grade 
level; information represented in codes for more 
or less detail and implied content was contained in 
the reviewer notes. 

The current study followed the NAEP pair-wise 
comparison approach but employed four sets 
of standards. One set — the American Diploma 
Project (ADP) — was designated as the bench- 
mark set. The standards statements of the three 
college readiness comparison standards sets — 
the ACT College Readiness Standards (ACT; 

ACT, Inc. 2007), College Board College Readiness 



FIGURE Al 

Pair-wise comparison methodology with the 
American Diploma Project standards set as the 
benchmark to which ACT, College Board, and 
Standards for Success were aligned, 2008 



American 

Diploma 

Project 

Total number of 
statements = 62 



American 

Diploma 

Project 

Total number of 
statements = 62 



American 

Diploma 

Project 

Total number of 
statements = 62 





Standards 
for Success 

Total number of 
statements = 73 



Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. 
(2007); College Board (2006); and Conley (2003). 
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• Partial alignment. Some portion (1-99 per- 
cent) of the content in the benchmark (ADP) 
standards statement aligns with some portion 
of the content in the comparison standards set 
(ACT, College Board, or S4S). 

• No alignment. None of the content in the 
benchmark (ADP) standards statement 
aligns with any of the content in the com- 
parison standards set (ACT, College Board, 
or S4S). 

Final alignments and ratings were determined 
during a consensus meeting with the senior 
reviewer. 

Examples of complete and partial statement 
alignments. Two examples of complete align- 
ment are provided in table Al. In example 1 the 



ADP standards statement completely aligns with 
two S4S statements considered together. In this 
example, the benchmark statement aligns with 
the comparison statements even though the 
wording is not identical. In example 2 the ADP 
statement completely aligns with the aggregate 
content of five statements from the College 
Board comparison set. In both cases the re- 
viewer notes explain the reasons for the rating of 
complete alignment. 

Two examples of partial alignment are provided in 
table A2. In example 1 the ADP statement partially 
aligns with three ACT statements. In example 2 
the ADP statement partially aligns with only one 
statement from the College Board comparison 
standards set. In both examples the reviewer 
notes explain the reason for the rating of partial 
alignment. 



TABLE Al 

Examples of complete alignment of the content of the American Diploma Program benchmark college 
readiness standards statements with the content of comparison standards sets, 2008 



Example 1 



Example 2 



Benchmark strand 
and statement 



Language A6. Recognize 
nuances in the meanings 
of words; choose words 
precisely to enhance 
communication. 



Literature H4. Analyze 
the setting, plot, theme, 
characterization and 
narration of classic and 
contemporary short stories 
and novels. 



Statements with complete alignment 
to the benchmark statement 



S4S I.B.3. Understand vocabulary and content, 
including subject-area terminology; connotative 
and denotative meanings; and idiomatic meanings. 
S4S II.D.5. Use words correctly; use words that 
mean what the writer intends to say; and use a 
varied vocabulary. 

CB R1.2.1 Uses understanding of setting and its 
connections to other narrative elements to guide 
comprehension of literary texts. 

CB R1.2.2 Uses understanding of plot and its 
connections to other narrative elements to guide 
comprehension of literary texts. 

CB R1.2.3 Uses understanding of characterization 
and its connections to other narrative elements to 
guide comprehension of literary texts. 

CB R1.2.4 Uses understanding of theme and its 
connections to other narrative elements to guide 
comprehension of literary texts. 

CB R1 .2.5 Uses understanding of narrative 
perspective and its connections to other narrative 
elements to guide comprehension of literary texts. 



Reviewer notes 



Connotative/denotative 
suggests recognizing 
nuances in words. This 
pushed the rating to 
complete alignment. 



In aggregate, these [College 
Board] statements provide 
a complete alignment to 
the ADP statement. 



Note: Statement identifier codes, such as A6, were used in the study to identify specific standard statements. The codes used to identify standards state- 
ments generally followed the prescribed coding format of each college readiness standards set, with some modifications. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); and Conley (2003). 
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TABLE A2 

Examples of partial alignment of the content of the American Diploma Program benchmark college 
readiness standards statements with the content of comparison standards sets, 2008 



Example 1 



Example 2 



Benchmark strand and statement 



Writing C2. Select and use formal, 
informal, literary or technical language 
appropriate for the purpose, audience 
and context of the communication. 



Logic E5. Recognize common logical 
fallacies, such as the appeal to pity 
(■ argumentum ad misericordiam), the 
personal attack ( argumentum ad 
hominem), the appeal to common 
opinion ( argumentum ad populum ) and 
the false dilemma (assuming only two 
options when there are more options 
available); understand why these 
fallacies do not prove the point being 
argued. 



Statements with partial alignment 
to the benchmark statement 



ACT E-3 24-27-3 Word Choice in 
Terms of Style, Tone, Clarity, and 
Economy: Use the word or phrase most 
appropriate in terms of the content of 
the sentence and tone of the essay 

ACT W-5 11-1 2-1 -b Using Language: 
Show effective use of language to 
clearly communicate ideas by using 
precise and varied vocabulary 

ACT W-5 09-10-1-b Using Language: 
Show competent use of language to 
communicate ideas by using some 
precise and varied vocabulary 

CB R3.1.2 Analyzes how an author 
creates an authorial persona, uses 
reasoning and evidence, and appeals 
to audience's emotions, interests, 
values, and beliefs to achieve specific 
purposes. 



Reviewer notes 



Use of technical language is 
not specifically mentioned 
in ACT. 



The [College Board] 
statement does not 
address all of the specific 
elements of logical fallacies 
described in the ADP 
statement. 



Note: Statement identifier codes, such as C2, were used in the study to identify specific standard statements. The codes used to identify standards state- 
ments generally followed the prescribed coding format of each college readiness standards set, with some modifications. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); and Conley (2003). 



Structure of content alignment tables. An example 
of how each content alignment table was struc- 
tured and populated is provided in figure A2. 

This study can be seen as three separate content 
alignment studies, using a methodology (pair-wise 
comparison of three sets to a single benchmark set) 
that is consistent with the parameters of the initial 
work conducted for the Commission for a Col- 
lege Ready Texas (comparison of the ACT, College 
Board, and S4S standards sets to ADP as the bench- 
mark). Although the research team independently 
aligned the three comparison sets of standards in 
the present study to the ADP benchmark, all results 
from the three pair-wise comparisons using ADP 
standards set as the benchmark are represented in 
a single alignment table (available on request and 



not reproduced here because of space limitations) 
instead of as separate results for each pair. The find- 
ings are also presented by strand in appendix C. The 
benchmark comparison methodology enables read- 
ers to see simultaneously which statements from the 
three comparison sets align to each ADP statement. 
Statements from ACT, College Board, and S4S that 
could not be aligned to any of the ADP statements 
are not presented in the alignment table but are 
provided in appendix D. 



Cognitive complexity rating methodology 

Cognitive complexity was assessed by compar- 
ing the distribution of standards statements from 
each set of standards across four levels of cognitive 
complexity (Webb 2002). Cognitive complexity 
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FIGURE A2 

Example of the structure of the full alignment table for the American Diploma Project benchmark standards 
set and the ACT comparison standards set, 2008 



American 
Diploma Project 
(ADP) Standard 
Statement 


American 
Diploma Project 


Content 

rating 

ACT Standard Statement (ACT to Reviewer notes 

E =English; W=Writing; R=Reading ADP) on alignment 


A. Language ^ 



A5. Identify 
the meaning of 
common idioms, 
as well as literary, 
classical and 
biblical allusions; 
use them in 
oral and written 
communication. 



A6. Recognize 
nuances in the 
meanings of 
words; choose 
words precisely 
to enhance 
communication. 



American 
Diploma Project 
statement 



R-4 28-32-1 
Meanings of Words: 
Determine the 
appropriate meaning 
of words, phrases, 
or statements 
from figurative or 
somewhat technical 
contexts 



E-5 33-36-1 

Conventions of Usage: 
Provide idiomatically 
and contextually 
appropriate 
prepositions 
following verbs in 
situations involving 
sophisticated 
language or ideas 

R-4 33-36-1 Meanings 
of Words: Determine, 
even when the 
language is richly 
figurative and the 
vocabulary is difficult, 
the appropriate 
meaning of 
context-dependent 
words, phrases, 
or statements in 
virtually any passage 



R-4 33-36-1 Meanings 
of Words: Determine, 
even when the 
language is richly 
figurative and the 
vocabulary is difficult, 
the appropriate 
meaning of 
context-dependent 
words, phrases, 
or statements in 
virtually any passage 



E-5 20-23-1 
Conventions 
of Usage: Use 
idiomatically 
appropriate 
prepositions, 
especially in 
combination with 
verbs (e.g., long for, 
appeal to) 




ACT statement(s) that 
show some alignment 
to the American 
Diploma Project strand 



E-3 24-27-3 Word 
Choice in Terms of 
Style, Tone, Clarity, 
and Economy: Use 
the word or phrase 
most appropriate in 
terms of the content 
of the sentence and 
tone of the essay 



W-5 11-1 2-1 -b Using 
Language: Show 
effective use of 
language to clearly 
communicate ideas 
by using precise and 
varied vocabulary 



Partial 

Alignment 




ACT does not refer 
to allusions in any 
standard. ACT does 
not address oral 
communication. 
Different levels 
within the same 
ACT strand indicate 
that the standard 
can be performed 
at various levels of 
competence. 



Content rating as 
determined by expert 
reviewers 



Complete 

Alignment 



These alignments 
dealt with revision, 
while E3 addressed 
the production of 
text. 



Expert 

reviewer 

comments 

i 



Note: Statement identifier codes, such as Language A5, were used in the study to identify specific standard statements. The codes used to identify ADP 
statements followed ADP's prescribed coding format; for example, "A" indicates a statement in the language strand and "5" indicates the fifth standard 
statement in that strand. The codes used to identify ACT statements were modified to ease their use in this study. The coding scheme included a number- 
letter combination that conveyed the score range and location of the standard statement in the ACT standards document. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); and Conley (2003). 



ratings were assigned to each statement by two in- 
dependent reviewers. Individual reviewers worked 
independently using Webb’s depth of knowledge 
(DoK) scale (2002) to rate the level of cognitive 
complexity of each statement: 



• Level 1— recall requires students to use simple 
skills or abilities to retrieve or recite facts. 

• Level 2 — skill/concept requires a level of com- 
prehension and subsequent processing across 
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portions of text to make inferences beyond 
simple recall or recitation of stated facts. 

• Level 3 — strategic thinking focuses on reason- 
ing, planning skills, making more complex 
inferences, and applying ideas from the text; 
students may be encouraged to explain, gener- 
alize, or connect ideas. 

• Level 4 — extended thinking requires inves- 
tigation and higher order thinking skills to 
process multiple solutions to a given problem. 

A more detailed description of the Webb DoK scale, 
including examples, is provided in appendix E. 

The cognitive complexity ratings of the two 
independent reviewers were discussed during 
consensus meetings held under the supervision of 
a senior reviewer, with final ratings determined by 
consensus at the meetings. 

A two -column cognitive complexity rating table 
was created for each standards set, with standards 
statements in the first column and the correspond- 
ing cognitive complexity level noted in the second 
column. An example of how each cognitive com- 
plexity rating table was structured and populated 
is provided in figure A3. 



Review process 

Throughout the review process, weekly progress 
meetings were held between the team managing 
the overall study— including the study design, 
implementation, analysis, and reporting (research 
team)— and the team conducting the content 
alignment and cognitive complexity ratings 
(review team). Also during these meetings, the 
review team provided any completed data tables to 
the research team for review. 

Step 1-selecting reviewers. The methodology of 
this study required ratings from two indepen- 
dent reviewers and a senior reviewer to supervise 
consensus discussions. English language arts 
teachers with experience in alignment studies 



were recruited as primary reviewers, and an expe- 
rienced researcher was selected as the supervising 
senior reviewer. More information about reviewer 
qualifications is provided in appendix B. 

Step 2-training reviewers. Before training, the 
two primary reviewers were provided with copies 
of the four sets of standards and asked to review 
the structure, organization, and content of each. 
Then the senior reviewer conducted a three-hour 
training session for the two primary reviewers, 
reviewing in detail the three-level content align- 
ment rating scale and the Webb (2002) cognitive 
complexity rating scale. The primary reviewers 
then independently practiced aligning and rating a 
small number of ADP statements with statements 
from ACT, College Board, and S4S. To conclude 
the training session, the primary reviewers and 
senior reviewer reconvened to discuss ratings and 
discrepancies related to the rating scales. 

Step 3-rating ADP cognitive complexity levels. As 

the first activity subsequent to training, reviewers 
individually rated each ADP statement on the cog- 
nitive complexity scale using the Webb DoK level 
descriptions (see appendix E). Making cognitive 
complexity rating the first activity ensured that 
reviewers carefully read and engaged with each 
statement before content alignment began. 

Step 4-achieving consensus on ADP cognitive com- 
plexity levels. After individually assigning cogni- 
tive complexity ratings to all ADP statements, the 
two independent reviewers met with the senior 
reviewer to compare ratings and achieve consen- 
sus where ratings differed. The role of the senior 
reviewer was to facilitate consensus and make the 
final decision if consensus could not be reached. 
Consensus meetings typically lasted about two 
hours. Once the cognitive complexity ratings were 
finalized, 5 percent of the statements that the re- 
viewers had rated independently were reviewed for 
rater drift (the tendency for reviewers or assessors 
to unintentionally redefine criteria over time). The 
check was conducted by having the reviewers in- 
dependently rate the selected statements again and 
compare the results with their original ratings. 11 
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FIGURE A3 

Example of the structure of the cognitive complexity rating table for American Diploma Project college 
readiness standards statements, 2008 



r 



American 



\ 




A1. Demonstrate control of standard English through the use of grammar, 
punctuation, capitalization and spelling. 



The emphasis is on standard 
English 



A2. Use general and specialized dictionaries, thesauruses and glossaries 
(print and electronic) to determine the definition, pronunciation, 
etymology, spelling and usage of words. 



A3. Use roots, affixes and cognates to determine the meaning of 
unfamiliar words. 

A4. Use context to determine the meaning of unfamiliar words. 



Cognitive complexity 
rating as determined 
by expert reviewers 



A5. Identify the meaning of common idioms, as well as literary, classical 
and biblical allusions; use them in oral and written communication. 



K 



Identify meaning" is at the level 
of skill/concept while "use them" 
(in oral and written form) gets 



A6. Recognize nuances in the met 
to enhance communication. 



American 
Diploma Project 
statement 



tse words precisely 



/ 


Expert 


closer to the application described 
in strategic thinking 




reviewer 






comments 




V 







A7. Comprehend and communicate quantitative, technical and 
mathematical information. 



It is possible that Webb's cognitive 
complexity rating scale does not 
address this area; "comprehend" 
could indicate skill/concept 



/Vote: Statement identifier codes, such as language A1, were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "A" indicates a statement in the Language strand and "1" indicates the first standard statement 
in that strand. 



a. Rating is based on Webb's (2002) cognitive complexity scale of 1 to 4 where 1 represents recall, 2 represents skill/concept, 3 represents strategic thinking, 
and 4 represents extended thinking. 

Source: Cognitive complexity summary reports from expert content reviewers (July 2008) drawing on college readiness standards statements in Achieve, 
Inc. (2004). 



Step 5-rating and achieving consensus on com- 
parison sets’ cognitive complexity levels. Reviewers 
individually rated each ACT statement using the 
cognitive complexity scale and then met with the 
senior reviewer to compare ratings and achieve 
consensus where ratings differed. After consen- 
sus was established, 5 percent of the statements 
that the reviewers had rated independently were 
reviewed for rater drift. This process was repeated 
first with College Board and then with S4S. The 
cognitive complexity ratings were conducted 
independent of the content alignment of the 
statements and rating of the level of content 
alignment. 



Step 6-comparison and alignment of ADP- ACT 
content. Using the ADP-ACT content alignment 
table and beginning with the first ADP statement 
in the first ADP strand, each reviewer inde- 
pendently and systematically searched all ACT 
statements for those containing content aligned 
to the ADP benchmark statement. This was an 
exhaustive search: all ACT statements with align- 
ing content were included. Once all completely and 
partially aligned ACT statements were identified, 
the reviewer assigned a content alignment rating 
to the ADP standard based on the cumulative con- 
tent of all the aligned ACT statements (complete 
alignment, partial alignment, or no alignment). 
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Consensus meetings between the independent 
reviewers and the senior reviewer were held after 
completion of every two ADP strands until all 
ADP statements were aligned and the content 
alignment levels were rated. Consensus meetings 
were held approximately every two weeks during 
this time. Once the ADP-ACT content alignment 
was completed and the content alignment levels 



were rated, 5 percent of the ADP statements were 
reviewed to check for rater drift. 

Steps 7 and 8-comparison and alignment of 
ADP-College Board and ADP-S4S content. The 

ADP-College Board and ADP-S4S content align- 
ments were conducted in the same manner as the 
ADP-ACT alignment. 
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APPENDIX B 

REVIEWER QUALIFICATIONS AND ROLES 

AND INTERRATER RELIABILITY 

This appendix provides more detail on interrater 
reliability, including information on reviewer 
qualifications. 

Reviewer qualifications and roles 

The review team consisted of a senior reviewer and 
two primary independent reviewers. The senior 
reviewer has a doctorate in English education and 
several years of experience designing and teach- 
ing English language arts courses for grades 9-12, 
13 years of experience teaching English language 
arts in the university setting, and several years of 
experience working with state education agencies. 
The two primary reviewers were secondary and 
postsecondary English language arts teachers who 
had previously participated in an English language 
arts alignment project using similar rating scales 
to align state high school standards to ACT and 
American Diploma Project (ADP) standards, using 
both as the benchmarks. The secondary school 
teacher was a reading specialist with a doctorate 
in reading education who has worked at the state 
and university levels in reading education. The 
postsecondary teacher holds a doctorate with a 
focus on reading education and has experience in 
developing reading assessments. 

The senior reviewer conducted initial training, 
monitored the progress of ratings, conducted 
consensus meetings, and served as the final arbiter 
if consensus on ratings could not be reached. The 
other two reviewers conducted the alignment and 
assigned the ratings. 



Interrater reliability: content alignment 

Standards alignment research is, by nature, a sub- 
jective process. Use of expert judgment is a critical 
element of that process. Multiple experts are used 
so that the unique perspective and knowledge of 
each individual contributes to results that gener- 
alize beyond one individual’s ratings. However, 



the use of multiple reviewers does not provide an 
advantage if there is little agreement. Low levels of 
reviewer agreement may indicate problems with 
the rating scales, qualifications of the review- 
ers, training, or other methodological decisions. 
Therefore, it is important to evaluate agreement 
among reviewers as an indicator of the quality of 
the research process and the potential generaliz- 
ability of the findings. 

The term interrater reliability refers to the meth- 
ods for summarizing the amount of agreement 
between multiple independent reviewers. Typi- 
cally, the higher the level of agreement, the more 
confident one can be that the assigned ratings 
would be replicated by others following the same 
procedures. Because this study employed two 
expert reviewers to make independent judgments 
using a subjective rating scale, a comparison of 
these independent ratings can provide informa- 
tion on initial consensus of the reviewers. How- 
ever, since the final ratings were determined using 
a consensus methodology, the initial agreement or 
disagreement is not critical to the validity of the 
final consensus ratings and alignment. 

Two approaches to summarizing interrater agree- 
ment are reported here: percent agreement and the 
intraclass correlation (table Bl). Percent agreement 
is useful because it is simply the proportion of 
identical ratings assigned by the two reviewers. 
However, this approach does not account for the 
possibility of agreement by chance, or ratings 
that are close but not an exact match. Therefore, a 
second method is also reported here, the intra- 
class correlation (Shrout and Fleiss 1979), which 
assumes that each reviewer brings measurement 
error into the rating process. The intraclass corre- 
lation also accounts for small discrepancies, such 
as when reviewer 1 rates a complete alignment and 
reviewer 2 rates a partial alignment. 



Interrater reliability: cognitive complexity 

Interrater reliability for cognitive complexity is 
reported in the same manner as for content align- 
ment, with two exceptions. The table of summary 
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statistics (table B2) contains all four standards 
sets. Cognitive complexity ratings were made 
for every statement within each set, regardless 
of whether statements aligned to any statements 
from the benchmark set. Note the relatively high 
agreement for ADP (75 percent), and the corre- 
sponding intraclass correlation. 



a. Overall percent agreement in independent alignment ratings prior to 
the consensus meeting for the 62 American Diploma Project benchmark 
statements. 

b. Calculated using SPSS, version 16.0 (SPSS, Inc. 2007) — two-way 
random effects model, absolute agreement, average measures. This is 
equivalent to Shrout and Fleiss (1979) Case 2, which assumes the two 
raters are drawn from a population of raters. This is also equivalent to an 
absolute G (phi) coefficient (Mushquash and O'Connor 2006, p. 543). 

Source: Expert reviewer activities (April-September 2008) drawing on 
standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); College 
Board (2006); Conley (2003). 



TABLE B1 

Content alignment interrater agreement prior to 
consensus meeting, 2008 



Comparison 


Percent 


Intraclass 


standards set 


agreement 3 


correlation 6 


ACT 


73 


0.78 


College Board 


48 


0.69 


Standards for Success 


69 


0.57 



TABLE B2 

Cognitive complexity interrater agreement prior to consensus meeting, 2008 



1 Standards set 


Number of statements 3 


Percent agreement 6 


Intraclass correlation 


American Diploma Project 


59 d 


75 d 


0.77 d 


ACT 


191 


46 


0.67 


College Board 


11 5/91 e 


54 e 


0.50 e 


Standards for Success 


73 


53 


0.62 



a. Cognitive complexity ratings were conducted for all statements in each standards set. 

b. This value represents a perfect match based on the four-point Webb (2002) depth of knowledge (DoK) scale and would therefore (other things being 
equal) tend to appear lower than in the three-level content alignment scale. 

c. Calculated using SPSS, version 16.0 (SPSS, Inc. 2007) — two-way random effects model, absolute agreement, average measures. This is equivalent to 
Shrout and Fleiss (1979) case 2, which assumes that the two raters are drawn from a population of raters. This is also equivalent to an absolute G (phi) coef- 
ficient (Mushquash and O'Connor 2006, p. 543). 

d. Statistics are based on paired ratings for 59 of 62 statements. Reviewer 1 did not assign ratings to 3 statements prior to the consensus meeting, due to 
uncertainty about how to apply the Webb DoK scale to "software presentations" and two statements about "explaining themes" and "demonstrating knowl- 
edge" of literature. These statements were discussed and consensus reached as with all other ratings. It cannot be known how the lack of three initial ratings 
might have affected final consensus ratings or agreement rates. 

e. Statistics for College Board are based on paired ratings for 91 of 115 statements. Reviewer 2 did not assign ratings to 24 statements prior to the consensus 
meeting. This reviewer was uncertain about how to apply the Webb DoK scale to College Board standards focused on oral communication and analysis of 
media. These statements were discussed and consensus reached as with all other ratings. It cannot be known how the lack of 24 initial ratings might have 
affected final consensus ratings or agreement rates. 

Source: Expert reviewer activities (April-September 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); College Board (2006); 
Conley (2003). 
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APPENDIX C 

CONTENT ALIGNMENT BY AMERICAN 
DIPLOMA PROJECT STRAND 

The results of all three independent alignments are 
represented, in detail, by the full alignment table. 
This information was abstracted into the eight 
summary tables — one for each of the eight Ameri- 
can Diploma Project (ADP) strands provided here. 



Language 

The ADP language strand contains seven state- 
ments. ACT has complete alignment to two 
statements in the ADP language strand, College 
Board has complete alignment to four statements, 
and Standards for Success (S4S) has complete 
alignment to four statements. ACT has partial 
alignment to three statements, and S4S has partial 
alignment to three statements. Finally, ACT has 
no alignment to two statements and College Board 
has no alignment to three statements (table Cl). 



TABLE Cl 

Alignment of American Diploma Project language strand statements with ACT, College Board, and Standards 
for Success statements, 2008 





Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Language 


A1. Demonstrate control of standard English through the use of grammar, punctuation, 
capitalization, and spelling. 


28 


2 


10 


A2. Use general and specialized dictionaries, thesauruses, and glossaries (print and 
electronic) to determine the definition, pronunciation, etymology, spelling, and usage 
of words. 


0 


0 


1 


A3. Use roots, affixes, and cognates to determine the meaning of unfamiliar words. 


0 


0 


2 


A4. Use context to determine the meaning of unfamiliar words. 


1 


1 


2 


A5. Identify the meaning of common idioms, as well as literary, classical, and biblical 
allusions; use them in oral and written communication. 


4 


3 


2 


A6. Recognize nuances in the meanings of words; choose words precisely to enhance 
communication. 


3 


5 


2 


A7. Comprehend and communicate quantitative, technical and mathematical 
information. 


2 


0 


4 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "A1,” were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "A" indicates a statement in the language strand and "1" indicates the first standard statement 
in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Communication 

The ADP communication strand contains seven 
statements. College Board has complete alignment 
to three statements and partial alignment to three 
statements. ACT and S4S have no alignment to any 
statements, and College Board has no alignment to 
one statement (table C2). 



TABLE C2 

Alignment of American Diploma Project communication strand statements with ACT, College Board, and 
Standards for Success statements, 2008 



Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Communication 


B1. Give and follow spoken instructions to perform specific tasks, to answer questions, 
or to solve problems. 


0 


0 


0 


B2. Summarize information presented orally by others. 


0 


2 


0 


B3. Paraphrase information presented orally by others. 


0 


2 


0 


B4. Identify the thesis of a speech and determine the essential elements that elaborate it. 


0 


3 


0 


B5. Analyze the ways in which the style and structure of a speech support or confound 
its meaning or purpose. 


0 


10 


0 


B6. Make oral presentations that exhibit a logical structure appropriate to the 
audience, context and purpose; group related ideas and maintain a consistent focus; 
include smooth transitions; support judgments with sound evidence and well-chosen 
details; make skillful use of rhetorical devices; employ proper eye contact, speaking 
rate, volume, enunciation, inflection, and gestures to communicate ideas effectively. 


0 


14 


0 


B7. Participate productively in self-directed work teams for a particular purpose (for 
example, to interpret literature, write or critique a proposal, solve a problem, make 
a decision), including posing relevant questions; listening with civility to the ideas of 
others; extracting essential information from others' input; building on the ideas of 
others and contributing relevant information or ideas in group discussions; consulting 
texts as a source of ideas; gaining the floor in respectful ways; defining individuals' 
roles and responsibilities and setting clear goals; acknowledging the ideas and 
contributions of individuals in the group; understanding the purpose of the team 
project and the ground rules for decision-making; maintaining independence of 
judgment, offering dissent courteously, ensuring a hearing for the range of positions 
on an issue, and avoiding premature consensus; tolerating ambiguity and a lack of 
consensus; and selecting leader/spokesperson when necessary. 


0 


14 


0 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "B1," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "B” indicates a statement in the communication strand and "1" indicates the first standard 
statement in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Writing 

The ADP writing strand contains 10 statements. 

ACT has complete alignment to one statement, while 
College Board and S4S each has complete alignment 



to four statements. ACT has partial alignment to 
four statements, while College Board and S4S each 
has partial alignment to five statements. ACT has no 
alignment to five statements, while College Board and 
S4S each has no alignment to one statement (table C3). 



TABLE C3 

Alignment of American Diploma Project writing strand statements with ACT, College Board, and Standards 
for Success statements, 2008 





Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Writing 


Cl. Plan writing by taking notes, writing informal outlines, and researching. 


0 


6 


3 


C2. Select and use formal, informal, literary, or technical language appropriate for the 
purpose, audience and context of the communication. 


3 


3 


6 


C3. Organize ideas in writing with a thesis statement in the introduction, well 
constructed paragraphs, a conclusion, and transition sentences that connect 
paragraphs into a coherent whole. 


11 


6 


3 


C4. Drawing on readers' comments on working drafts, revise documents to develop or 
support ideas more clearly, address potential objections, ensure effective transitions 
between paragraphs, and correct errors in logic. 


20 


2 


3 


C5. Edit both one's own and others' work for grammar, style, and tone appropriate to 
audience, purpose and context. 


33 


4 


2 


C6. Cite print or electronic sources properly when paraphrasing or summarizing 
information, quoting, or using graphics. 


0 


1 


1 


C7. Determine how, when, and whether to employ technologies (such as computer 
software, photographs, and video) in lieu of, or in addition to, written communication. 


0 


3 


2 


C8. Present written material using basic software programs (such as Word, Excel, and 
PowerPoint) and graphics (such as charts, ratios, and tables) to present information 
and ideas best understood visually. 


0 


0 


0 


C9. Write an academic essay (for example, a summary, an explanation, a description, 
a literary analysis essay) that develops a thesis; creates an organizing structure 
appropriate to purpose, audience, and context; includes relevant information and 
excludes extraneous information; makes valid inferences; supports judgments with 
relevant and substantial evidence and well-chosen details; and provides a coherent 
conclusion. 


15 


22 


7 


Cl 0. Produce work-related texts (for example, memos, e-mails, correspondence, 
project plans, work orders, proposals, bios) that address audience needs, stated 
purpose, and context; translate technical language into nontechnical English; include 
relevant information and exclude extraneous information; use appropriate strategies, 
such as providing facts and details, describing or analyzing the subject, explaining 
benefits or limitations, comparing or contrasting, and providing a scenario to illustrate; 
anticipate potential problems, mistakes, and misunderstandings that might arise 
for the reader; create predictable structures through the use of headings, white 
space, and graphics, as appropriate; and adopt a customary format, including proper 
salutation, closing, and signature, when appropriate. 


0 


22 


5 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "Cl," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "C" indicates a statement in the writing strand and "1" indicates the first standard statement in 
that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 




APPENDIX C. CONTENT ALIGNMENT BY AMERICAN DIPLOMA PROJECT STRAND 



27 



Research 

The ADP research strand contains five state- 
ments. College Board has complete alignment to 
one statement, and S4S has complete alignment 



to three statements. College Board has partial 
alignment to four statements, and S4S has partial 
alignment to one statement. ACT has no align- 
ment to any of the five statements, and S4S has no 
alignment to one statement (table C4). 



TABLE C4 

Alignment of American Diploma Project research strand statements with ACT, College Board, and Standards 
for Success statements, 2008 



Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Research 


D1. Define and narrow a problem or research topic. 


0 


2 


1 


D2. Gather relevant information from a variety of print and electronic sources, as well 
as from direct observation, interviews, and surveys. 


0 


3 


4 


D3. Make distinctions about the credibility, reliability, consistency, strengths, and 
limitations of resources, including information gathered from Web sites. 


0 


2 


3 


D4. Report findings within prescribed time and/or length requirements, as 
appropriate. 


0 


1 


0 


D5. Write an extended research essay (approximately 6 to 10 pages), building on 
primary and secondary sources, that marshals evidence in support of a clear thesis 
statement and related claims; paraphrases and summarizes with accuracy and 
fidelity the range of arguments and evidence supporting or refuting the thesis, as 
appropriate; and cites sources correctly and documents quotations, paraphrases, and 
other information using a standard format. 


0 


7 


11 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "D1," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "D" indicates a statement in the research strand and "1” indicates the first standard statement 
in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Logic 

The ADP logic strand contains nine statements. 
S4S has complete alignment to two statements, 
while College Board and S4S do not have complete 
alignment to any statements. ACT has partial 



alignment to three statements, while College 
Board and S4S each have partial alignment to six 
statements. ACT has no alignment to six state- 
ments, College Board has no alignment to three 
statements, and S4S has no alignment to one state 
ment (table C5). 



TABLE C5 

Alignment of American Diploma Project logic strand statements with ACT, College Board, and Standards for 
Success statements, 2008 





Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Logic 


El. Distinguish among facts and opinions, evidence, and inferences. 


5 


0 


1 


E2. Identify false premises in an argument. 


0 


3 


2 


E3. Describe the structure of a given argument; identify its claims and evidence; and 
evaluate connections among evidence, inferences, and claims. 


0 


5 


2 


E4. Evaluate the range and quality of evidence used to support or oppose an 
argument. 


10 


3 


5 


E5. Recognize common logical fallacies, such as the appeal to pity (argumentum ad 
misericordiam), the personal attack ( argumentum ad hominem), the appeal to common 
opinion ( argumentum ad populum ) and the false dilemma (assuming only two options 
when there are more options available); understand why these fallacies do not prove 
the point being argued. 


0 


1 


7 


E6. Analyze written or oral communications for false assumptions, errors, loaded 
terms, caricature, sarcasm, leading questions, and faulty reasoning. 


0 


10 


4 


E7. Understand the distinction between a deductive argument (where, if the premises 
are all true and the argument's form is valid, the conclusion is inescapably true) and 
inductive argument (in which the conclusion provides the best or most probable 
explanation of the truth of the premises, but is not necessarily true). 


0 


0 


1 


E8. Analyze two or more texts addressing the same topic to determine how authors 
reach similar or different conclusions. 


0 


0 


0 


E9. Construct arguments (both orally and in writing) that develop a thesis that 
demonstrates clear and knowledgeable judgment; structure ideas in a sustained 
and logical fashion; use a range of strategies to elaborate and persuade, such as 
descriptions, anecdotes, case studies, analogies, and illustrations; clarify and defend 
positions with precise and relevant evidence, including facts, expert opinions, 
quotations, and/or expressions of commonly accepted beliefs and logical reasoning; 
anticipate and address the reader's concerns and counterclaims; and provide clear and 
effective conclusions. 


5 


38 


9 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "El," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "E" indicates a statement in the logic strand and "1" indicates the first standard statement in 
that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Informational text 

The ADP informational text strand contains 11 
statements. ACT, College Board, and Standards 
for Success (S4S) each have complete alignment to 
two statements. ACT has partial alignment to four 



statements, College Board has partial alignment to 
seven statements, and S4S has partial alignment 
to five statements. ACT has no alignment to five 
statements, College Board has no alignment to 
two statements, and S4S has no alignment to four 
statements (table C6). 



TABLE C6 

Alignment of American Diploma Project informational text strand statements with ACT, College Board, and 
Standards for Success statements, 2008 





Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Informational text 


FI. Follow instructions in informational or technical texts to perform specific tasks, 
answer questions, or solve problems. 


0 


0 


1 


F2. Identify the main ideas of informational text and determine the essential elements 
that elaborate them. 


5 


1 


0 


F3. Summarize informational and technical texts and explain the visual components 
that support them. 


1 


6 


2 


F4. Distinguish between a summary and a critique. 


1 


0 


1 


F5. Interpret and use information in maps, charts, graphs, time lines, tables and 
diagrams. 


0 


2 


1 


F6. Identify interrelationships between and among ideas and concepts within a text, 
such as cause-and-effect relationships. 


17 


2 


2 


F7. Synthesize information from multiple informational and technical sources. 


0 


1 


0 


F8. Draw conclusions based on evidence from informational and technical texts. 


4 


1 


1 


F9. Analyze the ways in which a text's organizational structure supports or confounds 
its meaning or purpose. 


3 


1 


4 


F10. Recognize the use or abuse of ambiguity, contradiction, paradox, irony, 
incongruities, overstatement, and understatement in text and explain their effect on 
the reader. 


0 


1 


0 


F11. Evaluate informational and technical texts for their clarity, simplicity, and 
coherence and for the appropriateness of their graphics and visual appeal. 


0 


3 


0 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "FI," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "F" indicates a statement in the informational text strand and "1" indicates the first standard 
statement in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Media 



The ADP media strand contains four state- statements. ACT and S4S have no alignment to 

ments. College Board has complete alignment any statements, and College Board has no align- 

to one statement and partial alignment to two ment to one statement (table C7). 



TABLE C7 

Alignment of American Diploma Project media strand statements with ACT, College Board, and Standards 
for Success statements, 2008 



Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Media 


Gl. Evaluate the aural, visual, and written images and other special effects used 
in television, radio, film, and the Internet for their ability to inform, persuade, and 
entertain (for example, anecdote, expert witness, vivid detail, tearful testimony, and 
humor). 


0 


2 


0 


G2. Examine the intersections and conflicts between the visual (such as media images, 
painting, film, and graphic arts) and the verbal. 


0 


0 


0 


G3. Recognize how visual and sound techniques or design (such as special effects, 
camera angles, and music) carry or influence messages in various media. 


0 


2 


0 


G4. Apply and adapt the principles of written composition to create coherent media 
productions using effective images, text, graphics, music, and/or sound effects — if 
possible — and present a distinctive point of view on a topic (for example, PowerPoint 
presentations, videos). 


0 


6 


0 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "G1," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "G" indicates a statement in the media strand and "1" indicates the first standard statement in 
that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 
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Literature 

The ADP literature strand contains nine state- 
ments. College Board has complete alignment to 
two statements, S4S has complete alignment to four 
statements, and ACT has no complete alignments. 



ACT has partial alignment to two statements, Col- 
lege Board has partial alignment to four statements, 
and S4S has partial alignment to three statements. 
ACT has no alignment to seven statements, College 
Board has no alignment to three statements, and 
S4S has no alignment to two statements (table C8). 



TABLE C8 

Alignment of American Diploma Project literature strand statements with ACT, College Board, and Standards 
for Success statements, 2008 





Number and level of alignment 
of standards statements 


American Diploma Project strand and statements 


ACT 


College 

Board 


Standards 
for Success 


Literature 


HI. Demonstrate knowledge of 18th and 19th century foundational works of American 
literature. 


0 


0 


2 


H2. Analyze foundational U.S. documents for their historical and literary significance 
(for example, The Declaration of Independence, the Preamble to the U.S. Constitution, 
Abraham Lincoln's "Gettysburg Address," Martin Luther King's "Letter from 
Birmingham Jail"). 


0 


2 


1 


H3. Interpret significant works from various forms of literature: poetry, novel, 
biography, short story, essay, and dramatic literature; use understanding of genre 
characteristics to make deeper and subtler interpretations of the meaning of the text. 


0 


6 


3 


H4. Analyze the setting, plot, theme, characterization, and narration of classic and 
contemporary short stories and novels. 


7 


5 


4 


H5. Demonstrate knowledge of metrics, rhyme scheme, rhythm, alliteration, and other 
conventions of verse in poetry. 


0 


0 


0 


H6. Identify how elements of dramatic literature (for example, dramatic irony, 
soliloquy, stage direction, and dialogue) articulate a playwright's vision. 


0 


0 


0 


H7. Analyze works of literature for what they suggest about the historical period in 
which they were written. 


0 


2 


3 


H8. Analyze the moral dilemmas in works of literature, as revealed by characters' 
motivation and behavior. 


6 


1 


1 


H9. Identify and explain the themes found in a single literary work; analyze the ways in 
which similar themes and ideas are developed in more than one literary work. 


0 


2 


2 



Note: The darker shade represents complete alignment to the ADP statement, the lighter shade represents partial alignment, and no shade represents no 
alignment. Statement identifier codes, such as "HI," were used in the study to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's prescribed coding format; for example, "H" indicates a statement in the literature strand and "1" indicates the first standard statement 
in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004); ACT, Inc. (2007); 
College Board (2006); Conley (2003). 




32 



AMERICAN DIPLOMA PROJECT ENGLISH LANGUAGE ARTS COLLEGE READINESS STANDARDS 



APPENDIX D 

UNALIGNED STANDARDS 
STATEMENTS FROM BENCHMARK AND 
COMPARISON STANDARDS SETS 

This appendix contains tables showing American 
Diploma Project (ADP) standards statements that 



did not align with statements in any of the com- 
parison standards sets (table Dl) and statements 
from each of the comparison standards sets — the 
ACT College Readiness Standards (ACT), College 
Board College Readiness Standards, and Standards 
for Success (S4S)— that do not align to ADP (tables 
D2-D4). 



TABLE Dl 

American Diploma Project unique statements, by strand, 2008 



Strand and statements 



B. Communication 

B1. Give and follow spoken instructions to perform specific tasks, to answer questions, or to solve problems. 

C. Writing 

C8. Present written material using basic software programs (such as Word, Excel, and PowerPoint) and graphics (such as 
charts, ratios, and tables) to present information and ideas best understood visually. 

E. Logic 

E8. Analyze two or more texts addressing the same topic to determine how authors reach similar or different conclusions. 

G. Media 

G2. Examine the intersections and conflicts between the visual (such as media images, painting, film, and graphic arts) and the 
verbal. 

H. Literature 

H5. Demonstrate knowledge of metrics, rhyme scheme, rhythm, alliteration, and other conventions of verse in poetry. 

H6. Identify how elements of dramatic literature (for example, dramatic irony, soliloquy, stage direction, and dialogue) 
articulate a playwright's vision. 

Note: Statement identifier codes, such as "B1," were used in the study to identify specific standard statements. The codes used to identify American Diploma 
Project (ADP) statements followed ADP's prescribed coding format; for example, "B" indicates a statement in the communication strand and "1" indicates the 
first standard statement in that strand. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004). 



TABLE D2 

ACT statements that did not align to American Diploma Project statements, by strand, 2008 



Strand and statements 


Reading 


R-l Main ideas and author's approach 


13-15-1 


Recognize a clear intent of an author or narrator in uncomplicated literary narratives. 


16-19-1 


Identify a clear main idea or purpose of straightforward paragraphs in uncomplicated literary narratives. 


20-23-1 


Infer the main idea or purpose of straightforward paragraphs in uncomplicated literary narratives. 


20-23-2 


Understand the overall approach taken by an author or narrator (e.g., point of view, kinds of evidence used) in 
uncomplicated passages. 


24-27-1 


Identify a clear main idea or purpose of any paragraph or paragraphs in uncomplicated passages. 


24-27-2 


Infer the main idea or purpose of straightforward paragraphs in more challenging passages. 


24-27-3 


Summarize basic events and ideas in more challenging passages. 



(CONTINUED) 
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TABLE D2 (CONTINUED) 

ACT statements that did not align to American Diploma Project statements, by strand, 2008 



Strand and statements 


24-27-4 


Understand the overall approach taken by an author or narrator (e.g., point of view, kinds of evidence used) in more 
challenging passages. 


R-2 Supporting details 


13-15-1 


Locate basic facts (e.g., names, dates, events) clearly stated in a passage. 


16-19-1 


Locate simple details at the sentence and paragraph level in uncomplicated passages. 


16-19-2 


Recognize a clear function of a part of an uncomplicated passage. 


20-23-1 


Locate important details in uncomplicated passages. 


20-23-2 


Make simple inferences about how details are used in passages. 


24-27-1 


Locate important details in more challenging passages. 


24-27-2 


Locate and interpret minor or subtly stated details in uncomplicated passages. 


28-32-1 


Locate and interpret minor or subtly stated details in more challenging passages. 


R-3 Sequential, comparative, and cause-and-effect relationships 


13-15-1 


Determine when (e.g., first, last, before, after) or if an event occurred in uncomplicated passages. 


13-15-2 


Recognize clear cause-effect relationships described within a single sentence in a passage. 


16-19-1 


Identify relationships between main characters in uncomplicated literary narratives. 


16-19-2 


Recognize clear cause-effect relationships within a single paragraph in uncomplicated literary narratives. 


20-23-1 


Order simple sequences of events in uncomplicated literary narratives. 


20-23-2 


Identify clear relationships between people, ideas, and so on in uncomplicated passages. 


20-23-3 


Identify clear cause-effect relationships in uncomplicated passages. 


24-27-1 


Order sequences of events in uncomplicated passages. 


24-27-2 


Understand relationships between people, ideas, and so on in uncomplicated passages. 


24-27-3 


Identify clear relationships between characters, ideas, and so on in more challenging literary narratives. 


24-27-4 


Understand implied or subtly stated cause-effect relationships in uncomplicated passages. 


28-32 -1 


Order sequences of events in more challenging passages. 


R-4 Meanings of words 


13-15-1 


Understand the implication of a familiar word or phrase and of simple descriptive language. 


16-19-1 


Use context to understand basic figurative language. 


20-23-1 


Use context to determine the appropriate meaning of some figurative and nonfigurative words, phrases, and 
statements in uncomplicated passages. 


24-27-1 


Use context to determine the appropriate meaning of virtually any word, phrase, or statement in uncomplicated 
passages. 


24-27-2 


Use context to determine the appropriate meaning of some figurative and nonfigurative words, phrases, and 
statements in more challenging passages. 


R-5 Generalizations and conclusions 


13-15-1 


Draw simple generalizations and conclusions about the main characters in uncomplicated literary narratives. 


16-19-1 


Draw simple generalizations and conclusions about people, ideas, and so on in uncomplicated passages. 


20-23-1 


Draw generalizations and conclusions about people, ideas, and so on in uncomplicated passages. 


20-23-2 


Draw simple generalizations and conclusions using details that support the main points of more challenging 
passages. 


24-27-1 


Draw subtle generalizations and conclusions about characters, ideas, and so on in uncomplicated literary narratives. 



(CONTINUED) 





34 



AMERICAN DIPLOMA PROJECT ENGLISH LANGUAGE ARTS COLLEGE READINESS STANDARDS 



TABLE D2 (CONTINUED) 

ACT statements that did not align to American Diploma Project statements, by strand, 2008 



Strand and statements 


English 


E-1 Topic development in terms of purpose and focus 


16-19-1 


Identify the basic purpose or role of a specified phrase or sentence. 


16-19-2 


Delete a clause or sentence because it is obviously irrelevant to the essay. 


20-23-1 


Identify the central idea or main topic of a straightforward piece of writing. 


24-27-1 


Identify the focus of a simple essay, applying that knowledge to add a sentence that sharpens that focus or to 
determine if an essay has met a specified goal. 


24-27-2 


Delete material primarily because it disturbs the flow and development of the paragraph. 


24-27-3 


Add a sentence to accomplish a fairly straightforward purpose such as illustrating a given statement. 


E-2 Organization, unity and coherence 


13-15-1 


Use conjunctive adverbs or phrases to show time relationships in simple narrative essays (e.g., then, this time). 


16-19-1 


Select the most logical place to add a sentence in a paragraph. 


20-23-1 


Use conjunctive adverbs or phrases to express straightforward logical relationships (e.g., first, afterward, in response). 


20-23-3 


Add a sentence that introduces a simple paragraph. 


24-27-3 


Add a sentence to introduce or conclude the essay or to provide a transition between paragraphs when the essay is 
fairly straightforward. 


E-3 Word choice in terms of style, tone, clarity, and economy 


20-23-1 


Delete redundant material when information is repeated in different parts of speech (e.g., "alarmingly startled"). 


20-23-3 


Determine the clearest and most logical conjunction to link clauses. 


24-27-2 


Identify and correct ambiguous pronoun references. 


28-32 -1 


Correct redundant material that involves sophisticated vocabulary and sounds acceptable as conversational English 
(e.g., "an aesthetic viewpoint" versus "the outlook of an aesthetic viewpoint"). 


E-4 Sentence structure and formation 


16-19-1 


Determine the need for punctuation and conjunctions to avoid awkward-sounding sentence fragments and fused 
sentences. 


24-27-2 


Maintain consistent verb tense and pronoun person on the basis of the preceding clause or sentence. 


28-32 -2 


Maintain a consistent and logical use of verb tense and pronoun person on the basis of information in the paragraph 
or essay as a whole. 


Writing 


W-1 Expressing judgments 


03-4-1 


Show a little understanding of the persuasive purpose of the task but neglect to take or to maintain a position on the 
issue in the prompt. 


03-4-2 


Show limited recognition of the complexity of the issue in the prompt. 


05-6-1 


Show a basic understanding of the persuasive purpose of the task by taking a position on the issue in the prompt but 
may not maintain that position. 


05-6-2 


Show a little recognition of the complexity of the issue in the prompt by acknowledging, but only briefly describing, a 
counterargument to the writer's position. 


07-8-1 


Show understanding of the persuasive purpose of the task by taking a position on the issue in the prompt. 


07-8-2-a 


Show some recognition of the complexity of the issue in the prompt by acknowledging counterarguments to the 
writer's position. 


07-8-2-b 


Show some recognition of the complexity of the issue in the prompt by providing some response to 
counterarguments to the writer's position. 



(CONTINUED) 




APPENDIX D. UNALIGNED STANDARDS STATEMENTS FROM BENCHMARK AND COMPARISON STANDARDS SETS 



35 



TABLE D2 (CONTINUED) 

ACT statements that did not align to American Diploma Project statements, by strand, 2008 



Strand and statements 


09-10-2-a 


Show recognition of the complexity of the issue in the prompt by partially evaluating implications and/or 
complications of the issue. 


09-10-2-b Show recognition of the complexity of the issue in the prompt by posing and partially responding to 
counterarguments to the writer's position. 


W-2 Focusing on the topic 


03-4-1 


Maintain a focus on the general topic in the prompt through most of the essay. 


05-6-1 


Maintain a focus on the general topic in the prompt throughout the essay. 


07-8-1 


Maintain a focus on the general topic in the prompt throughout the essay and attempt a focus on the specific issue 
in the prompt. 


W-3 Developing a position 


03-4-1 


Offer a little development, with one or two ideas; if examples are given, they are general and may not be clearly 
relevant; resort often to merely repeating ideas. 


03-4-2 


Show little or no movement between general and specific ideas or examples. 


05-6-1 


Offer limited development of ideas using a few general examples; resort sometimes to merely repeating ideas. 


05-6-2 


Show little movement between general and specific ideas and examples. 


07-8-1 


Develop ideas by using some specific reasons, details, and examples. 


07-8-2 


Show some movement between general and specific ideas or examples. 


W-4 Organizing ideas 


03-4-1 


Provide a discernible organization with some logical grouping of ideas in parts of the essay. 


03-4-2 


Use a few simple and obvious transitions. 


03-4-3 


Present a discernible, though minimally developed, introduction and conclusion. 


05-6-1 


Provide a simple organization with logical grouping of ideas in parts of the essay. 


05-6-2 


Use some simple and obvious transitional words, though they may at times be inappropriate or misleading. 


05-6-3 


Present a discernible, though underdeveloped, introduction and conclusion. 


07-8-1 


Provide an adequate but simple organization with logical grouping of ideas in parts of the essay but with little 
evidence of logical progression of ideas. 


07-8-2 


Use some simple and obvious, but appropriate, transitional words and phrases. 


07-8-3 


Present a discernible introduction and conclusion with a little development. 


09-10-1 


Provide unity and coherence throughout the essay, sometimes with a logical progression of ideas. 


09-10-2 


Use relevant, though at times simple and obvious, transitional words and phrases to convey logical relationships 
between ideas. 


09-10-3 


Present a somewhat developed introduction and conclusion. 


W-5 Using language 


03-4-1 -a 


Show limited control of language by correctly employing some of the conventions of standard English grammar, 
usage, and mechanics, but with distracting errors that sometimes significantly impede understanding. 


03-4-1 -b 


Show limited control of language by using simple vocabulary. 


03-4-1 -c 


Show limited control of language by using simple sentence structure. 


05-6-1-a 


Show a basic control of language by correctly employing some of the conventions of standard English grammar, 
usage, and mechanics, but with distracting errors that sometimes impede understanding. 


05-6-1-b 


Show a basic control of language by using simple but appropriate vocabulary. 


05-6-1-c 


Show a basic control of language by using a little sentence variety, though most sentences are simple in structure. 
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TABLE D2 (CONTINUED) 

ACT statements that did not align to American Diploma Project statements, by strand, 2008 



Strand and statements 



07-8-1 -a Show adequate use of language to communicate by correctly employing many of the conventions of standard 
English grammar, usage, and mechanics, but with some distracting errors that may occasionally impede 
understanding. 

07-8-1-b Show adequate use of language to communicate by using appropriate vocabulary. 

07-8-1 -c Show adequate use of language to communicate by using some varied kinds of sentence structures to vary pace. 

09-1 0-1 -a Show competent use of language to communicate ideas by correctly employing most conventions of standard 
English grammar, usage, and mechanics, with a few distracting errors but none that impede understanding. 

09-1 0-1 -c Show competent use of language to communicate ideas by using several kinds of sentence structures to vary pace 
and to support meaning. 

/Vote: The codes used to identify ACT statements followed ACT's prescribed coding format but were modified to ease their use in this study. The coding 
scheme included a number-letter combination that conveyed the score range and location of the standard statement in the ACT standards document. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in ACT, Inc. (2007). 



TABLE D3 

College Board statements that did not align to American Diploma Standards statements, by strand, 2008 

Speaking 

1: Understanding the communication process objective 

S1.1 Student understands the transactional nature of the communication process. 

51.1.1 Understands the transactional nature and components of the communication process, including 
speaker, listener, message, channel, feedback, and noise. 

51.1.2 Understands how speaker's and listener's internal variables affect communication. 

51.1.3 Understands how contextual variables affect communication. 

3: Preparing and Delivering Presentations Objectives 

S3.4 Student presents, monitors audience engagement, and adapts delivery. 

S3.4.2 Monitors audience feedback; makes inferences about audience engagement, understanding, and 
agreement; and adjusts delivery and content to achieve purposes and goals. 

Reading 

1: Comprehension of words, sentences, and components of texts objectives 
R1.1 Student comprehends the meaning of words and sentences. 

R1.1.1 Uses the origins, history, and evolution of words and concepts to enhance understanding. 

R1.1.3 Integrates word meaning, grammar, syntax, and context to construct a coherent understanding of 
sections of text. 

2: Using prior knowledge, context, and understanding of language to comprehend and elaborate the meaning of texts objectives 
R2.1 Student uses prior knowledge to comprehend and elaborate the meaning of texts. 

R2.1 .2 Uses prior knowledge and experiences to extend and elaborate the meaning of texts. 

R2.3 Student uses knowledge of the evolution, diversity, and effects of language to comprehend and elaborate the 
meaning of texts. 

R2.3.1 Uses knowledge of the evolution and diversity of language to guide comprehension of texts. 

4: Using strategies to comprehend texts objectives 

R4.1 Student uses strategies to prepare to read. 
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TABLE D3 (CONTINUED) 

College Board statements that did not align to American Diploma Standards statements, by strand, 2008 



Strand and statements 




R4.1.1 


Identifies purposes and goals for reading to guide the reading process. 




R4.1.2 


Uses pre-reading strategies to develop expectations about the text and to guide the reading process. 


R4.2 


Student uses strategies to interpret the meaning of words, sentences, and ideas in texts. 




R4.2.1 


Uses text-focused strategies (e.g., re-reading, paraphrasing, chunking, close reading) to better 
understand texts, improve global understanding, and infer implied meanings of the text. 




R4.2.2 


Marks and annotates texts and takes notes during or after reading to identify and elaborate key ideas. 




R4.2.3 


Makes intentional bridging inferences and connections back to previous sentences and ideas across 
larger sections of text to resolve problems in comprehension. 




R4.2.4 


Uses text structures to make connections among ideas and improve comprehension. 


R4.3 


Student uses strategies to go beyond the text. 




R4.3.1 


Uses questions of self, author, text, and context to clarify and extend comprehension of texts. 




R4.3.2 


Uses think-aloud and self-explanation to extend and elaborate the meaning of the text. 




R4.3.3 


Uses visualization to represent and make connections among objects, setting, characters, events, 
processes, and concepts in texts. 




R4.3.4 


Uses a variety of primary and secondary sources to expand and deepen the understanding of texts. 


R4.5 


Student monitors comprehension and reading strategies throughout the reading process. 




R4.5.1 


Monitors comprehension while reading by generating questions to determine level of understanding, 
by participating in discussions about the text, by noting points of misunderstanding, and by trying to 
establish connections among ideas in the text and to prior knowledge. Adjusts reading strategies to 
improve comprehension. 




R4.5.2 


Assesses post-reading comprehension, memory, and learning and adjusts reading strategies to improve 
comprehension. 


Listening 


1: Understanding the communication process objective 


LI .1 


Student understands the transactional nature of the communication process. 




LI .1.1 


Understands the transactional nature and components of the communication process, including 
speaker, listener, message, channel, feedback, and noise. 




LI .1.2 


Understands how speaker's and listener's internal variables affect communication. 




LI .1.3 


Understands how contextual variables affect communication. 


2: Managing barriers to listening objective 


L2.1 


Student manages barriers to listening. 




L2.1.1 


Recognizes his or her own internal variables that can pose barriers to effective listening and uses a 
variety of strategies to manage them. 




L2.1.2 


Understands that language represents and constructs how listeners perceive events, people, groups, 
and ideas and that it has both positive and negative implications that can affect listeners in different 
ways. 




L2.1.3 


Recognizes that external variables can pose barriers to effective listening and uses a variety of strategies 
to prevent or overcome them. 


3: Listening for diverse purposes objectives 


L3.1 


Student listens to comprehend. 




L3.1.4 


Uses a variety of response strategies to clarify explicit and implicit meanings of messages. 


L3.3 


Student listens empathically. 
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TABLE D3 (CONTINUED) 

College Board statements that did not align to American Diploma Standards statements, by strand, 2008 



Strand and statements 




L3.3.1 


Uses a variety of mental and physical strategies to focus attention on the speaker, the speaker's message, 
and the speaker's emotions in order to listen empathically. 




L3.3.4 


Uses a variety of verbal and nonverbal strategies to respond to the speaker's message in order to 
indicate support, keep the speaker talking, and build understanding and empathy. 


Media literacy 


1: Understanding the nature of media objective 


M1.1 


Student understands the nature of media communication. 




Ml. 1.2 


Understands how media producers capture, measure, and interpret responses to media messages as 
indicators of the messages' effectiveness and how media producers use this feedback to modify media 
messages. 


2: Understanding, interpreting, analyzing, and evaluating media communication objective 


M2.1 


Student understands, interprets, analyzes, and evaluates media communication. 




M2.1.1 


Analyzes how media producers use conventional production elements to achieve specific effects. 




M2.1.2 


Analyzes how media producers use production elements and techniques to establish narrative elements 
(e.g., setting, mood, tone, character, plot) and create specific effects. 




M2.1.3 


Analyzes how the media channel and production elements affect the targeted audience, achieve the 
purpose, and convey the media producer's point of view. 




M2.1.4 


Recognizes how his or her prior knowledge, experiences, attitudes, beliefs, and demographic 
characteristics, as well as the context, affect the interpretation of a media message. 


3: Composing and producing media communication objectives 


M3. 3 


Student evaluates and revises a media communication. 




M3. 3. 2 


Recognizes the power of media communication and the importance of using media ethically. Explains 
the role of legal regulations and fair use policies when setting purposes and goals, developing content, 
and publishing a media communication. 



Note: The codes used to identify College Board statements followed College Board's prescribed coding format of standard, standard number, objective, and 
performance expectation number. For example, S1.1.1 indicates speaking standard 3, objective 1, and performance expectation 1. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in College Board (2006). 



TABLE D4 

Standards for Success statements that did not align to American Diploma Program statements, by strand, 2008 



Strand and statements 



I. Reading and comprehension 

I.A. Successful students employ reading skills and strategies to understand literature. They: 

I.A.7. * recognize and comprehend narrative terminology and techniques, such as author versus narrator, stated 

versus implied author and historical versus present-day reader. 

I.B. Successful students use reading skills and strategies to understand informational texts. They: 

I.B.2. use monitoring and self correction, as well as reading aloud, as means to ensure comprehension. 

I.C. Successful students are able to understand the defining characteristics of texts and to recognize a variety of literary 
forms and genres. They: 

I.C. 2. understand the formal constraints of different types of texts and can distinguish between, for example, a 
Shakespearean sonnet and a poem written in free verse. 
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TABLE D4 (CONTINUED) 

Standards for Success statements that did not align to American Diploma Program statements, by strand, 2008 



Strand and statements 



I.C.6. use aesthetic qualities of style, such as diction or mood, as a basis to evaluate literature that contains 
ambiguities, subtleties or contradictions. 

l.D. Successful students are familiar with a range of world literature. They: 

I.D.2. demonstrate familiarity with authors from literary traditions beyond the English speaking world. 

I. E. Successful students are able to discuss with understanding the relationships between literature and its historical and 

social contexts. They: 

I. E.4. are able to discuss with understanding the relationships between literature and politics, including the political 

assumptions underlying an author's work and the impact of literature on political movements and events. 

II. Writing 

II. D. Successful students use writing conventions to write clearly and coherently. They: 

II. D.6. *demonstrate development of a controlled yet unique style and voice in writing where appropriate. 

II. E. Successful students use writing to communicate ideas, concepts, emotions and descriptions to the reader. They: 

II.E.1. know the difference between a topic and a thesis. 

IV. Critical thinking skills 

IV. A. Successful students demonstrate connective intelligence. They: 

IV. A.l. are able to discuss with understanding how personal experiences and values affect reading comprehension and 
interpretation. 

IV. A. 2. * demonstrate an ability to make connections between the component parts of a text and the larger theoretical 
structures, including presupposition, audience, purpose, writer's credibility or ethos, types of evidence or 
material being used and style. 

IV. B. Successful students demonstrate the ability to think independently. They: 

IV. B.l. are comfortable formulating and expressing their own ideas. 

* Denote items expected of students who plan to major in these fields of study (Conley 2003, p. 11). 

/Vote: The codes used to identify Standards for Success (S4S) statements followed S4S's prescribed coding format of pattern of knowledge foundation, skill, 
and skill number. For example, I.A.7 indicates knowledge foundation reading and comprehension, Successful students employ reading skills and strategies 
to understand literature, and skill 7. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Conley (2003). 
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APPENDIX E 

WEBB'S COGNITIVE COMPLEXITY 

LEVEL DESCRIPTIONS 

The following cognitive complexity level descrip- 
tions for reading and writing were taken verbatim 
from Webb’s Cognitive Complexity Criteria: Lan- 
guage Arts Levels for Depth of Knowledge (2002, pp. 
1-3) and used for initial training of reviewers. Both 
the reading and writing scales are based on the four 
levels described earlier in this report: level 1-recall, 
level 2-skill/concept, level 3-strategic thinking, 
and level 4-extended thinking. Reviewers in the 
current study used the appropriate scale based on 
the statement content. Subsequent consensus meet- 
ings among review team members refined how this 
language and terminology was interpreted during 
the rating process. Examples of statements from the 
four sets of college readiness standards in this study 
that reviewers rated at each Webb depth of knowl- 
edge (DoK) scale level are provided in tables E1-E4. 



Level 1 (Webb 2002, pp. 1 and 2) 

Reading. Level 1 ( Recall ) requires students to 
receive or recite facts or to use simple skills or abili- 
ties. Oral reading that does not include analysis of 
the text, as well as basic comprehension of a text is 
included. Items require only a shallow understand- 
ing of text presented and often consist of verbatim 
recall from text or simple understanding of a single 
word or phrase. Some examples that represent but 
do not constitute all of level 1 performance are: 



• Support ideas by reference to details in the text. 

• Use a dictionary to find the meaning of words. 

• Identify figurative language in a reading 
passage. 

Writing. Level 1 (Recall) requires the student 
to write or recite simple facts. This writing or 
recitation does not include complex synthesis or 
analysis but basic ideas. The students are engaged 
in listing ideas or words as in a brainstorming 
activity prior to written composition, are engaged 
in a simple spelling or vocabulary assessment, 
or are asked to write simple sentences. Students 
are expected to write and speak using standard 
English conventions. This includes using ap- 
propriate grammar, punctuation, capitalization, 
and spelling. Some examples that represent but 
do not constitute all of level 1 performance follow 
(table El): 

• Use punctuation marks correctly. 

• Identify standard English grammatical struc- 
tures and refer to resources for correction. 



Level 2 (Webb 2002, pp. 1 and 2-3) 

Reading. Level 2 (Skill/Concept) includes the 
engagement of some mental processing beyond 
recalling or reproducing a response; it requires 
both comprehension and subsequent processing 



TABLE El 

Examples of standards statements rated at cognitive complexity level 1 



Standards set 


Statement 

identifier 


Statement 


American Diploma Project 


A1 


Demonstrate control of standard English through the use of grammar, 
punctuation, capitalization and spelling 


ACT 


R-2 13-15-1 


Supporting Details: Locate basic facts (e.g., names, dates, events) clearly stated 
in a passage 


College Board 


W5.1 


Edits for conventions of standard written English and usage 


Standards for Success 


I.F.1. 


Identify the primary elements of the types of charts, graphs and visual media 
that occur most commonly in texts 



Source: Achieve, Inc. 2004; ACT, Inc. 2007; College Board 2006; Conley 2003. 
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of text or portions of text. Intersentence analysis 
of inference is required. Some important concepts 
are covered but not in a complex way. Standards 
and items at this level may include words such as 
summarize, interpret, infer, classify, organize, col- 
lect, display, compare, and determine whether fact 
or opinion. Literal main ideas are stressed. A level 
2 assessment item may require students to apply 
some of the skills and concepts that are covered in 
level 1. Some examples that represent but do not 
constitute all of level 2 performance are: 

• Use context cues to identify the meaning of 
unfamiliar words. 

• Predict a logical outcome based on informa- 
tion in a reading selection. 

• Identify and summarize the major events in a 
narrative. 

Writing. Level 2 (Skill/Concept) requires some 
mental processing. At this level students are 
engaged in first draft writing or brief extempora- 
neous speaking for a limited number of purposes 
and audiences. Students are beginning to connect 
ideas using a simple organizational structure. For 
example, students may be engaged in note-taking, 
outlining, or simple summaries. Text may be 
limited to one paragraph. Students demonstrate a 
basic understanding and appropriate use of such 
reference materials as a dictionary, thesaurus, or 



web site. Some examples that represent but do 
not constitute all of level 2 performance follow 
(table E2): 

• Construct compound sentences. 

• Use simple organizational strategies to struc- 
ture written work. 

• Write summaries that contain the main idea 
of the reading selection and pertinent details. 



3 (Webb 2002, pp. 1-3) 

Reading. Deep knowledge becomes more of a focus 
at Level 3 ( Strategic Thinking). Students are en- 
couraged to go beyond the text; however, they are 
still required to show understanding of the ideas 
in the text. Students may be encouraged to ex- 
plain, generalize, or connect ideas. Standards and 
items at level 3 involve reasoning and planning. 
Students must be able to support their thinking. 
Items may involve abstract theme identification, 
inference across an entire passage, or students’ 
application of prior knowledge. Items may also 
involve more superficial connections between 
texts. Some examples that represent but do not 
constitute all of level 3 performance are: 

• Determine the author’s purpose and describe 
how it affects the interpretation of a reading 
selection. 



TABLE E2 

Examples of standards statements rated at cognitive complexity level 2 



Standards set 


Statement 

identifier 


Statement 


American Diploma Project 


A3 


Use roots, affixes and cognates to determine the meaning of unfamiliar words 


ACT 


R-1 16-19-1 


Main Ideas and Author's Approach: Identify a clear main idea or purpose of 
straightforward paragraphs in uncomplicated literary narratives 


College Board 


R4.1.2 


Uses pre-reading strategies to develop expectations about the text and to 
guide the reading process 


Standards for Success 


I.A.2. 


Make supported inferences and draw conclusions based on textual features, 
seeking such evidence in text, format, language use, expository structures and 
arguments used 



Source: Achieve, Inc. 2004; ACT, Inc. 2007; College Board 2006; Conley 2003. 
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• Summarize intormation trom multiple Level 4 (Webb 2002, pp. 2 and 3) 



sources to address a specific topic. 

• Analyze and describe the characteristics of 
various types of literature. 

Writing. Level 3 ( Strategic Thinking) requires 
some higher level mental processing. Students 
are engaged in developing compositions that 
include multiple paragraphs. These compositions 
may include complex sentence structure and 
may demonstrate some synthesis and analysis. 
Students show awareness of their audience and 
purpose through focus, organization, and the use 
of appropriate compositional elements. The use 
of appropriate compositional elements includes 
such things as addressing chronological order in a 
narrative or including supporting facts and details 
in an informational report. At this stage students 
are engaged in editing and revising to improve the 
quality of the composition. Some examples that 
represent but do not constitute all of level 3 perfor- 
mance follow (table E3): 

• Support ideas with details and examples. 

• Use voice appropriate to the purpose and 
audience. 

• Edit writing to produce a logical progression 
of ideas. 



Reading. ITigher-order thinking is central and 
knowledge is deep at Level 4 ( Extended Thinking). 
The standard or assessment item at this level will 
probably be an extended activity, with extended 
time provided. The extended time period is not a 
distinguishing factor if the required work is only 
repetitive and does not require applying signifi- 
cant conceptual understanding and higher-order 
thinking. Students take information from at least 
one passage and are asked to apply this informa- 
tion to a new task. They may also be asked to 
develop hypotheses and perform complex analyses 
of the connections among texts. Some examples 
that represent but do not constitute all of level 4 
performance are: 

• Analyze and synthesize information from 
multiple sources. 

• Examine and explain alternative perspectives 
across a variety of sources. 

• Describe and illustrate how common themes 
are found across texts from different cultures. 

Writing. Higher-level thinking is central to level 
4 ( Extended Thinking). The standard at this level 
is a multi-paragraph composition that demon- 
strates synthesis and analysis of complex ideas 



TABLE E3 

Examples of standards statements rated at cognitive complexity level 3 



Standards set 


Statement 

identifier 


Statement 


American Diploma Project 


D3 


Make distinctions about the credibility, reliability, consistency, strengths and 
limitations of resources, including information gathered from Web sites 


ACT 


W-4 03-4-1 


Organizing Ideas: Provide a discernible organization with some logical 
grouping of ideas in parts of the essay 


College Board 


R4.3.1 


Uses questions of self, author, text, and context to clarify and extend 
comprehension of texts 


Standards for Success 


I.E.3. 


Demonstrate familiarity with the concept of the relativity of all historical 
perspectives, including their own 



Source: Achieve, Inc. 2004; ACT, Inc. 2007; College Board 2006; Conley 2003. 
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or themes. There is evidence of a deep awareness 
of purpose and audience. For example, informa- 
tional papers include hypotheses and support- 
ing evidence. Students are expected to create 
compositions that demonstrate a distinct voice 
and that stimulate the reader or listener to con- 
sider new perspectives on the addressed ideas 



and themes. An example that represents but 
does not constitute all of level 4 performance is 
(table E4): 

• Write an analysis of two selections, identify- 
ing the common theme and generating a 
purpose that is appropriate for both. 



TABLE E4 

Examples of standards statements rated at cognitive complexity level 4 



Standards set 


Statement 

identifier 


Statement 


American Diploma Project 


E8 


Analyze two or more texts addressing the same topic to determine how 
authors reach similar or different conclusions 


ACT 


W-2 11-12-2 


Focusing on the Topic: Present a critical thesis that clearly establishes the focus 
on the writer's position on the issue 


College Board 


W3.1.3 


Uses rhetorical appeals and organizational structures to establish a credible 
voice 


Standards for Success 


III. A. 3. 


Identify claims in their writing that require outside support or verification 



Source: Achieve, Inc. 2004; ACT, Inc. 2007; College Board 2006; Conley 2003. 
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APPENDIX F 

COGNITIVE COMPLEXITY BY STRAND FOR ALL 
FOUR COLLEGE READINESS STANDARDS SETS 

This appendix discusses cognitive complexity 
ratings by strand for the four college readiness 
standards sets. 



American Diploma Project cognitive complexity 

Variability in cognitive complexity was observed 
across the eight American Diploma Project 
(ADP) strands (figure FI). Overall, more than 
a quarter of the content in seven of the eight 
strands (language was the exception) was rated 
at cognitive complexity level 3-strategic think- 
ing. ffowever, the strands vary greatly in repre- 
sentations of the other three complexity levels. 
For example, level 1-recall is represented only in 
the language strand (14 percent). Level 2-skill/ 
concept is not represented by either media or 
literature strands but has 71 percent represen- 
tation in language. Finally, the highest level of 
cognitive complexity is missing from both the 



language and writing strands, with the greatest 
representations of level 4-extended thinking 
displayed in communication (29 percent) and 
media (25 percent). 



ACT cognitive complexity 

Level 3-strategic thinking is also well repre- 
sented in ACT, with the highest representation 
in English and writing, while the majority of the 
reading strand is represented by level 2-skill/ 
concept (figure F2). Compared with the other 
standards sets, ACT strands display the highest 
percentage of level 1-recall, and also the low- 
est percentage of level 4-extended thinking. 

One reason may be that wording in the ACT 
strands is very detailed to facilitate ACT test 
item development. This fact may make it difficult 
to assess some of the more abstract constructs 
described under level 4-extending thinking, 
which results in the lowest percentage of level 4 
cognitive complexity ratings on the Webb (2002) 
depth of knowledge (DoK) scale among the four 
standards sets. 



FIGURE FI 

Percentage of American Diploma Project standards statements at each level of cognitive complexity, 
by strand, 2008 

Percent ■ Level 1-recall 




Language Communication Writing Research Logic Informational Media Literature All American 

text Diploma Project 

American Diploma Project strands 



Note: Some components do not sum to 100 percent because of rounding. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in Achieve, Inc. (2004). 
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FIGURE F2 

Percentage of ACT standards statements at each 
level of cognitive complexity, by strand, 2008 



Percent ■ Level 1-recall 




English Writing Reading All ACT 

ACT strands 

Note: Some components do not sum to 100 percent because of rounding. 

Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in ACT, Inc. (2007). 



College Board cognitive complexity 

The majority of all five strands within the Col- 
lege Board standards set are represented by level 

3- strategic thinking (figure F3). In contrast, 
however, four of the strands represent level 

4- extended thinking, though each at less than 10 
percent. Also, only two of the strands represent 
level 1-recall complexity. Level 2-skill/concept 

is represented by each of the five strands, with a 
range of 13 percent (media literacy) to 39 percent 
(reading). 



Standards for Success cognitive complexity 

Standards for Success (S4S) displays the most 
even distribution of cognitive complexity across 
strands (figure F4). For example, S4S is the only 
set of standards that does not display strand 
averages above 70 percent for any one cogni- 
tive complexity level. It is also the only set of 
standards that displays a greater than 30 percent 
representation from level 4-extended thinking in 



FIGURE F3 

Percentage of College Board standards statements at each level of cognitive complexity, by strand, 2008 



Percent ■ Level 1 -recall 




Speaking Writing Reading Media Listening All 

Literacy College Board 



College Board strands 



Note: Some components do not sum to 100 percent because of rounding. 

Source: Alignment summary ratings from expert content reviewers (July 2008) drawing on standards statements in College Board (2006). 



46 



AMERICAN DIPLOMA PROJECT ENGLISH LANGUAGE ARTS COLLEGE READINESS STANDARDS 



any strand. Although the S4S standards state- 
ments were the most evenly distributed across 
the four levels, the distributions within each 
S4S strand vary. For example, more than 30 
percent of two strands that might be expected 
to show higher cognitive complexity levels 
(research skills and critical thinking skills) are at 
level 4-extended thinking. However, more than 
25 percent of the writing strand statements are 
at level 1-recall, and more than 30 percent of the 
reading and comprehension statements are at 
level 2-skill/concept. 



FIGURE F4 

Percentage of Standards for Success standards 
statements at each level of cognitive complexity, 
by strand, 2008 



Writing Reading and Research Critical All Standards 

comprehension skills thinking for Success 

skills 

Standards for Success strands 

A/ofe: Some components do not sum to 100 percent because of rounding. 

Source: Alignment summary ratings from expert content reviewers (July 
2008) drawing on standards statements in Conley (2003). 



Percent ■ Level 1-recall 

100 ■ Level 2-skill/concept 

Level 3-strategic thinking 
■ Level 4-extended thinking 
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APPENDIX G 

ALTERNATE CONTENT ALIGNMENT 

METHODOLOGIES 

Standards alignment research is typically con- 
ducted to evaluate the alignment between test 
items and assessment or curriculum standards 
(for example, Webb, Herman, and Webb 2007). In 
such studies assessment items are first matched 
to the relevant standards statements, and then 
judgments are made about how appropriately the 
test items measure the knowledge and skills in- 
tended by the standard. For alignment studies in 
general, one document serves as the benchmark 
against which other documents are aligned and 
evaluated. 

Less common in the alignment literature are com- 
parisons between sets of standards, but several 
such studies have been conducted. A recent series 
of studies aligned the math and science assess- 
ment standards of states in the Southwest Region 
to the most recent National Assessment of Edu- 
cational Progress (NAEP) assessment standards 
(Shapley and Brite 2008; Timms et al. 2007). These 
studies used the NAEP assessment standards as 
benchmarks to which state assessment standards 
were aligned. The level of alignment was then 
rated. These studies, as in nearly all alignment 
studies, compared only two documents (a pair- 
wise comparison). 

The research questions in the current study 
required comparing four sets of college readiness 
standards. The technical assistance research that 
was the genesis of the current study also com- 
pared four sets of college readiness standards and 
provided results in a single content alignment 
table. The single content alignment table enabled 
readers to determine at a glance the benchmark 
content that appears in most or all of the compari- 
son standards sets as well as content unique to the 
benchmark standards set. As a result, the ability 
to provide results in a single table was a priority 
when evaluating possible methodologies for the 
current study. 



The following four methodological approaches 
were considered: 

1. Perform a pair-wise comparison of all pos- 
sible pairs of standards, requiring either 6 or 
12 separate alignment pairings (depending 
on whether the direction of alignment is a 
concern). 

2. Use an external benchmark, such as a list of 
standards statements from another source. 

3. Allow the standards to form a content align- 
ment table inductively. In other words, there 
would be no single benchmark; a row would 
be formed whenever distinctly new content 
appeared in any of the standards. Content 
common to all standards would appear as a 
full row, while content unique to one set of 
standards would appear in a row with only 
a single cell filled. Raters would derive their 
own row and cell structure, which would then 
be resolved across raters. 12 

4. Adapt the alignment methodology (a pair- 
wise comparison) described in the NAEP sci- 
ence series (Timms et al. 2007) to compare a 
benchmark set of college readiness standards 
with three comparison sets of standards. 

Approach 1 (a pair-wise comparison of all possible 
pairs of standards), while possibly the most rigor- 
ous, would produce 6 or 12 pair-wise comparisons; 
these results could not be represented in a single 
content alignment table. While policymakers would 
have been interested in a simultaneous comparison 
of all four sets of college readiness standards, the 
usefulness of the final results was a more important 
consideration. In addition, a second expert judg- 
ment process would be needed to summarize and 
categorize the findings because of the lack of com- 
mon content categories across all four sets. 

Approach 2 (use of an external benchmark) intro- 
duces another set of statements, requiring a more 
universal and validated set of college readiness 
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content statements; no such framework currently 
exists for college readiness, and creating one was 
well beyond the scope of this study. 

Approach 3 (allowing the statements to form a 
content alignment table inductively), while in- 
triguing, introduces another layer of subjectivity. 
The inductively determined content benchmarks 
would need to be agreed on prior to alignment and 
would be less likely to be replicable. 



Approach 4 (pair-wise comparison to a single 
benchmark) was chosen for this study to take 
advantage of an alignment methodology already 
approved by the Institute for Education Sciences 
(Timms et al. 2007). This method, with adapta- 
tions, is described in detail in the body of this 
report and allows the results of the three pair-wise 
comparisons using American Diploma Project as 
the benchmark to be represented in a single align- 
ment table for ease of use. 



NOTES 
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NOTES 

1 . 



2 . 



3. 

4. 



A literature search was conducted to verify that 
no other sets of college readiness standards, 
intended as national standards, were avail- 
able. All permutations of two or more of the 
terms college, readiness, and standards were 
used to search databases (ERIC, EBSCO, and 
PSYCH-INFO), publications (Education Week 
and Chronicle of Higher Education), and state 
education web sites (through Google searches). 
In addition, interviews were conducted with 
the experts who provided testimony to the 
CCRT and a representative from the Fordham 
Foundation, which is well known for its stan- 
dards work (S. Stotsky, Representative, Ford- 
ham Foundation — personal communication, 
August 16, 2008). College readiness standards 
developed for use in a single state (for example, 
in Washington; Transition Mathematics Project 
2006), postsecondary standards not primarily 
intended for college readiness (for example, 
Partnership for 21st Century Schools n.d.), and 
national standards primarily designed as K-12 
standards (for example, National Assessment 
of Educational Progress; National Assessment 
Governing Board 2007; and National Council of 
Teachers of Mathematics 2000) were excluded 
from the study. No additional national college 
readiness standards were identified. 

Texas adopted new English language arts col- 
lege readiness standards in April 2008 (Texas 
Higher Education Coordinating Board 2008), 
and while the CCRT has been dissolved, 
college readiness standards remain a focus 
for the postsecondary and K-12 education 
commissioners. 

At the time this study was initiated, Texas was 
focused on revising its English language arts 
standards, so this study continued that focus. 

Rothman (2004) provides a thorough review 
of the dimensions that researchers have used 
over the past 10 years to evaluate content. 



5. The use of multiple raters is common prac- 
tice to improve the reliability and validity of 
subjective ratings (for example, Donner and 
Eliasziw 1987; Saito et al. 2006; Tinsley and 
Weiss 1975). 

6. See appendix G for a description of the other 
methodological approaches considered for 
this study. 

7. The literature search conducted as part of the 
initial study was replicated and confirmed 
that no other college readiness standards had 
been developed for national use between the 
initial and current studies. 

8. How each standards set categorizes and labels 
content differs; for this study the term strand 
is used to refer to a category, and the terms 
standards statements and statements are used 
interchangeably to denote the specific knowl- 
edge and skills in a category. 

9. The degree to which rater subjectivity may 
have affected the results of this study is no 
greater than for any other alignment study. 

10. The Timms et al. (2007) studies aligned the 
science domains of the 2009 NAEP assess- 
ment standards and state K-12 assessment 
standards; the Shapley and Brite studies 
(2008) aligned the mathematics domains of 
these same sets of standards. 

11. Rater drift is the tendency for raters or assessors 
to unintentionally redefine criteria over time. 
Rater drift checks were conducted several times 
during the review process to verify that there 
were no such shifts in criteria. Because drift 
occurred so infrequently (zero to one occur- 
rence per check), instances were not formally 
recorded. In the current study the consensus 
meetings served to continually recalibrate the 
reviewers’ understanding of the rating scales to 
the original definition. Therefore, the mini- 
mal rater drift that occurred did not influence 
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the final consensus ratings for either content 
alignment or cognitive complexity ratings. 

12. There is precedent for this approach. Wash- 
ington State developed a similar English 
language arts matrix when developing its 



college readiness standards (Washington 
Higher Education Coordinating Board 2007). 
However no final report, alignment ratings, or 
notes were documented, and no description of 
the research methodology (including number 
of raters) was reported. 



NOTES 
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